SDMX fetchers

This page explains how to build a downloader and a converter for providers that distribute data in SDMX format (Statistical Data and Metadata eXchange).

Why SDMX is different

SDMX is a standardised format for statistical data exchange, unlike the ad-hoc CSV or JSON APIs that some providers expose. Because the format is standard, the toolbox can ship base classes that do most of the work:

  • BaseSdmxDownloader downloads global structures, iterates the dataflow catalogue, and groups dataset files automatically

  • BaseSdmxConverter reads SDMX structures, extracts dimensions and attributes, converts series and observations, and builds a category tree

The fetcher author only needs to plug in the provider-specific HTTP endpoints and URLs. Compared to a regular CSV fetcher, an SDMX fetcher ends up with much less custom code — the base class handles the SDMX complexity.

BaseSdmxDownloader

BaseSdmxDownloader is a specialization of BaseDownloader for SDMX providers. It does three main things:

  • it downloads global SDMX structures such as DataFlow, Categorisation, and CategoryScheme

  • it downloads per-dataset SDMX resources grouped by dataflow

  • it stores SDMX files in a conventional directory layout with an SdmxDirManager

How it works

BaseSdmxDownloader expects you to implement three abstract methods:

  • _create_global_structure_resource(self, structure_type)

  • _create_dataset_data_resource(self, resource_id, *, dataflow, target_file)

  • _create_dataset_structures_resource(self, resource_id, *, dataflow, target_file)

Using those hooks, it builds the SDMX download pipeline.

The downloader uses an SdmxDirManager to determine where files belong inside source_data_dir:

  • get_dataset_dir(dataflow_id)

  • get_dataset_data_file(dataflow_id)

  • get_dataset_structure_file(dataflow_id)

  • load_dataflows() to parse global metadata and list available dataflows

It also groups each dataset download in a ResourceGroup so that the dataset data file and its related structure file are downloaded together.

Default global structures

If you do not override global_sdmx_structures, the downloader downloads the following structures by default:

  • StructureType.DataFlow

  • StructureType.Categorisation

  • StructureType.CategoryScheme

These files are often required to build SDMX category trees and interpret dataset metadata.

Download flow

BaseSdmxDownloader._download() does:

  1. _download_global_structures()

  2. _download_datasets()

Each dataset is created from a dataflow returned by _iter_dataflows(). By default, _iter_dataflows() loads dataflows from the SDMX directory and filters them by agency_id if one is provided.

Optional dataset freshness

_find_dataset_last_update(dataflow) can be overridden to return the dataset update timestamp. If provided, this timestamp is used to compute whether the dataset should be considered stale.

BaseSdmxConverter

BaseSdmxConverter is a specialization of BaseConverter for SDMX providers. It is designed to convert a directory of SDMX files into the DBnomics data model.

What it does

BaseSdmxConverter provides an SDMX-aware conversion pipeline:

  • it caches global SDMX structures before conversion with _sdmx_dir_manager.cache_global_structures({StructureType.DataFlow})

  • it creates a category tree converter from SDMX content with _create_category_tree_converter()

  • it creates a dataset converter per dataflow with _create_dataset_converter()

This means you usually do not need to enumerate dataset IDs manually: the converter reads them from the prepared SDMX dataset directory.

SDMX dataset iteration

The converter iterates datasets using sdmx_dir_manager.iter_dataflow_ids_from_datasets_dir(). For each dataflow ID, it loads the corresponding Dataflow object and creates a DatasetConverter.

Custom dataset converter

By default, BaseSdmxConverter._create_dataset_converter() returns an SdmxDatasetConverter. If your provider needs a custom conversion path, override this method and return your own converter class.

Strict mode for SDMX references

In SDMX, entities reference each other through refs that carry three fields: id, agency_id, and version. When the converter resolves these references (e.g. a dimension pointing to its codelist, or a dataflow pointing to its data structure), it must decide how strictly to match.

By default, the converter is lenient: it only matches on id and ignores agency_id and version. If no entity is found, the error is logged and the conversion continues — the affected labels or values are simply left empty.

You can tighten this behaviour with strict mode flags. When enabled, each flag requires an exact match on id, agency_id, and version. If no entity satisfies all three, an ArtefactNotFound exception is raised.

Available flags

There are four flags, each controlling a different kind of reference:

Flag

Class

Controlled reference

strict_structure_refs

SdmxModelOptions

Dataflow → DataStructure

strict_codelist_refs

SdmxModelOptions

Dimension / Attribute → Codelist

strict_concept_identity_refs

SdmxModelOptions

Dimension / Attribute → ConceptScheme

strict_dataflow_refs

SdmxCategoryTreeConverter

Categorisation → Dataflow

The first three belong to SdmxModelOptions and affect dataset-level conversion. The last one belongs to SdmxCategoryTreeConverter and affects the category tree build.

Error handling

In SdmxDatasetConverter, ArtefactNotFound exceptions are caught and logged — they do not halt the converter. The affected dimension or attribute ends up with no label and no values, but the rest of the dataset is still converted.

In SdmxCategoryTreeConverter, a missing dataflow is logged and ignored in non-strict mode. In strict mode, the exception propagates and the category tree build fails.

Enabling strict mode in a fetcher

Overrides BaseSdmxConverter._create_dataset_converter() and _create_category_tree_converter():

from dbnomics_toolbox.sdmx_utils.v2_1.processors.sdmx_dataset_converter import (
    SdmxDatasetConverter,
    SdmxModelOptions,
)
from dbnomics_toolbox.sdmx_utils.v2_1.processors.sdmx_category_tree_converter import (
    SdmxCategoryTreeConverter,
)


class MySdmxConverter(BaseSdmxConverter):
    def _create_dataset_converter(self, dataflow):
        return SdmxDatasetConverter(
            dataflow=dataflow,
            provider_code=self.provider_code,
            sdmx_dir_manager=self._sdmx_dir_manager,
            sdmx_model_options=SdmxModelOptions(
                strict_codelist_refs=True,
                strict_concept_identity_refs=True,
                strict_structure_refs=True,
            ),
        )

    def _create_category_tree_converter(self):
        return SdmxCategoryTreeConverter(
            sdmx_dir_manager=self._sdmx_dir_manager,
            strict_dataflow_refs=True,
        )

SDMX downloader example structure

A minimal SDMX downloader typically looks like this:

class MySdmxDownloader(BaseSdmxDownloader):
    def _create_global_structure_resource(self, structure_type: StructureType) -> FileResource:
        return SdmxResource(
            id=str(structure_type),
            url=self._build_structure_url(structure_type),
            target_file=self._sdmx_dir_manager.get_global_structure_file(structure_type),
        )

    def _create_dataset_data_resource(
        self,
        resource_id: str,
        *,
        dataflow: Dataflow,
        target_file: Path,
    ) -> FileResource:
        return SdmxResource(
            id=resource_id,
            url=self._build_data_url(dataflow.id),
            target_file=target_file,
        )

    def _create_dataset_structures_resource(
        self,
        resource_id: str,
        *,
        dataflow: Dataflow,
        target_file: Path,
    ) -> FileResource:
        return SdmxResource(
            id=resource_id,
            url=self._build_structure_url_for_dataflow(dataflow.id),
            target_file=target_file,
        )

The exact HTTP resource type depends on your provider, but the important part is that the downloader produces:

  • one or more global structure files

  • one dataset data file per dataflow

  • one dataset structure file per dataflow

SDMX converter example structure

A minimal SDMX converter is usually very small:

class MySdmxConverter(BaseSdmxConverter):
    def __init__(self, *, provider_code, source_data_dir, converted_data_writer, sdmx_dir_manager, **kwargs):
        super().__init__(
            provider_code=provider_code,
            source_data_dir=source_data_dir,
            converted_data_writer=converted_data_writer,
            sdmx_dir_manager=sdmx_dir_manager,
            **kwargs,
        )

In most cases, you only need to pass the right sdmx_dir_manager and let the base class handle the rest.