# SDMX fetchers

This page explains how to build a downloader and a converter for providers that distribute data in [SDMX format](https://sdmx.org/) (Statistical Data and Metadata eXchange).

## Why SDMX is different

SDMX is a standardised format for statistical data exchange, unlike the ad-hoc CSV or JSON APIs that some providers expose.
Because the format is standard, the toolbox can ship base classes that do most of the work:

- [](#BaseSdmxDownloader) downloads global structures, iterates the dataflow catalogue, and groups dataset files automatically
- [](#BaseSdmxConverter) reads SDMX structures, extracts dimensions and attributes, converts series and observations, and builds a category tree

The fetcher author only needs to plug in the provider-specific HTTP endpoints and URLs.
Compared to a regular CSV fetcher, an SDMX fetcher ends up with much less custom code — the base class handles the SDMX complexity.

## BaseSdmxDownloader

[](#BaseSdmxDownloader) is a specialization of [](#BaseDownloader) for SDMX providers.
It does three main things:

- it downloads global SDMX structures such as `DataFlow`, `Categorisation`, and `CategoryScheme`
- it downloads per-dataset SDMX resources grouped by dataflow
- it stores SDMX files in a conventional directory layout with an [](#SdmxDirManager)

### How it works

[](#BaseSdmxDownloader) expects you to implement three abstract methods:

- `_create_global_structure_resource(self, structure_type)`
- `_create_dataset_data_resource(self, resource_id, *, dataflow, target_file)`
- `_create_dataset_structures_resource(self, resource_id, *, dataflow, target_file)`

Using those hooks, it builds the SDMX download pipeline.

The downloader uses an [](#SdmxDirManager) to determine where files belong inside `source_data_dir`:

- `get_dataset_dir(dataflow_id)`
- `get_dataset_data_file(dataflow_id)`
- `get_dataset_structure_file(dataflow_id)`
- `load_dataflows()` to parse global metadata and list available dataflows

It also groups each dataset download in a [](#ResourceGroup) so that the dataset data file and its related structure file are downloaded together.

### Default global structures

If you do not override `global_sdmx_structures`, the downloader downloads the following structures by default:

- `StructureType.DataFlow`
- `StructureType.Categorisation`
- `StructureType.CategoryScheme`

These files are often required to build SDMX category trees and interpret dataset metadata.

### Download flow

[](#BaseSdmxDownloader)`._download()` does:

1. `_download_global_structures()`
2. `_download_datasets()`

Each dataset is created from a dataflow returned by `_iter_dataflows()`.
By default, `_iter_dataflows()` loads dataflows from the SDMX directory and filters them by `agency_id` if one is provided.

### Optional dataset freshness

`_find_dataset_last_update(dataflow)` can be overridden to return the dataset update timestamp.
If provided, this timestamp is used to compute whether the dataset should be considered stale.

## BaseSdmxConverter

[](#BaseSdmxConverter) is a specialization of [](#BaseConverter) for SDMX providers.
It is designed to convert a directory of SDMX files into the DBnomics data model.

### What it does

[](#BaseSdmxConverter) provides an SDMX-aware conversion pipeline:

- it caches global SDMX structures before conversion with `_sdmx_dir_manager.cache_global_structures({StructureType.DataFlow})`
- it creates a category tree converter from SDMX content with `_create_category_tree_converter()`
- it creates a dataset converter per dataflow with `_create_dataset_converter()`

This means you usually do not need to enumerate dataset IDs manually: the converter reads them from the prepared SDMX dataset directory.

### SDMX dataset iteration

The converter iterates datasets using `sdmx_dir_manager.iter_dataflow_ids_from_datasets_dir()`.
For each dataflow ID, it loads the corresponding [Dataflow](#dbnomics_toolbox.sdmx_utils.v2_1.model.dataflows.Dataflow) object and creates a [](#DatasetConverter).

### Custom dataset converter

By default, [](#BaseSdmxConverter)`._create_dataset_converter()` returns an [](#SdmxDatasetConverter).
If your provider needs a custom conversion path, override this method and return your own converter class.

### Strict mode for SDMX references

In SDMX, entities reference each other through **refs** that carry three fields: `id`, `agency_id`, and `version`.
When the converter resolves these references (e.g. a dimension pointing to its codelist, or a dataflow pointing to its data structure), it must decide how strictly to match.

By default, the converter is **lenient**: it only matches on `id` and ignores `agency_id` and `version`.
If no entity is found, the error is logged and the conversion continues — the affected labels or values are simply left empty.

You can tighten this behaviour with **strict mode flags**.
When enabled, each flag requires an exact match on `id`, `agency_id`, *and* `version`.
If no entity satisfies all three, an `ArtefactNotFound` exception is raised.

#### Available flags

There are four flags, each controlling a different kind of reference:

| Flag | Class | Controlled reference |
|------|-------|---------------------|
| `strict_structure_refs` | `SdmxModelOptions` | `Dataflow` → `DataStructure` |
| `strict_codelist_refs` | `SdmxModelOptions` | Dimension / Attribute → `Codelist` |
| `strict_concept_identity_refs` | `SdmxModelOptions` | Dimension / Attribute → `ConceptScheme` |
| `strict_dataflow_refs` | `SdmxCategoryTreeConverter` | `Categorisation` → `Dataflow` |

The first three belong to [](#SdmxModelOptions) and affect dataset-level conversion.
The last one belongs to [](#SdmxCategoryTreeConverter) and affects the category tree build.

#### Error handling

In [](#SdmxDatasetConverter), `ArtefactNotFound` exceptions are **caught and logged** — they do not halt the converter.
The affected dimension or attribute ends up with no label and no values, but the rest of the dataset is still converted.

In [](#SdmxCategoryTreeConverter), a missing dataflow is logged and ignored in non-strict mode.
In strict mode, the exception propagates and the category tree build fails.

#### Enabling strict mode in a fetcher

Overrides [](#BaseSdmxConverter)`._create_dataset_converter()` and `_create_category_tree_converter()`:

```python
from dbnomics_toolbox.sdmx_utils.v2_1.processors.sdmx_dataset_converter import (
    SdmxDatasetConverter,
    SdmxModelOptions,
)
from dbnomics_toolbox.sdmx_utils.v2_1.processors.sdmx_category_tree_converter import (
    SdmxCategoryTreeConverter,
)


class MySdmxConverter(BaseSdmxConverter):
    def _create_dataset_converter(self, dataflow):
        return SdmxDatasetConverter(
            dataflow=dataflow,
            provider_code=self.provider_code,
            sdmx_dir_manager=self._sdmx_dir_manager,
            sdmx_model_options=SdmxModelOptions(
                strict_codelist_refs=True,
                strict_concept_identity_refs=True,
                strict_structure_refs=True,
            ),
        )

    def _create_category_tree_converter(self):
        return SdmxCategoryTreeConverter(
            sdmx_dir_manager=self._sdmx_dir_manager,
            strict_dataflow_refs=True,
        )
```

## SDMX downloader example structure

A minimal SDMX downloader typically looks like this:

```python
class MySdmxDownloader(BaseSdmxDownloader):
    def _create_global_structure_resource(self, structure_type: StructureType) -> FileResource:
        return SdmxResource(
            id=str(structure_type),
            url=self._build_structure_url(structure_type),
            target_file=self._sdmx_dir_manager.get_global_structure_file(structure_type),
        )

    def _create_dataset_data_resource(
        self,
        resource_id: str,
        *,
        dataflow: Dataflow,
        target_file: Path,
    ) -> FileResource:
        return SdmxResource(
            id=resource_id,
            url=self._build_data_url(dataflow.id),
            target_file=target_file,
        )

    def _create_dataset_structures_resource(
        self,
        resource_id: str,
        *,
        dataflow: Dataflow,
        target_file: Path,
    ) -> FileResource:
        return SdmxResource(
            id=resource_id,
            url=self._build_structure_url_for_dataflow(dataflow.id),
            target_file=target_file,
        )
```

The exact HTTP resource type depends on your provider, but the important part is that the downloader produces:

- one or more global structure files
- one dataset data file per dataflow
- one dataset structure file per dataflow

## SDMX converter example structure

A minimal SDMX converter is usually very small:

```python
class MySdmxConverter(BaseSdmxConverter):
    def __init__(self, *, provider_code, source_data_dir, converted_data_writer, sdmx_dir_manager, **kwargs):
        super().__init__(
            provider_code=provider_code,
            source_data_dir=source_data_dir,
            converted_data_writer=converted_data_writer,
            sdmx_dir_manager=sdmx_dir_manager,
            **kwargs,
        )
```

In most cases, you only need to pass the right `sdmx_dir_manager` and let the base class handle the rest.