Base classes

Class diagram

        classDiagram
    BaseResource <|-- FileResource
    BaseResource <|-- ResourceGroup
    FileResource <|-- HttpResource
    HttpResource <|-- SdmxResource
    ResourceGroup <|-- BasePageResourceGroup
    ResourceGroup <|-- BasePartitionGroup

    class BaseResource {
        <<abstract>>
        +id: ResourceId
        +dataset_ids: frozenset[str] | None
        +updated_at: datetime | None
        +_start()
    }

    class FileResource {
        <<abstract>>
        +target_file: Path
        +keep: bool
        +optional: bool
        +_reformat_file()
        +_validate_mimetype()
        +_write_target_file(retry_state)
    }

    class HttpResource {
        +request: Request | str | URL
        +session: Session | None
        +dump_response: DumpResponse
        +timeout: Timeout | Timeouts | None
    }

    class SdmxResource {
        +normalize_sdmx_file_header: bool
        +raise_for_sdmx_error(error)
    }

    class ResourceGroup {
        +resources: list[FileResource]
        +target_dir: Path | None
        +_move_files_to_target_dir()
        +_write_target_file()
    }

    class BasePageResourceGroup {
        <<abstract>>
        +start_page_num: int
        +_create_resource(page_num) FileResource
    }

    class BasePartitionGroup {
        <<abstract>>
        +initial_partitions: list[TPartition]
        +_create_resource(partition, ...) FileResource
    }
    

BaseResource (abstract) is the base class for all resources. It defines the id, type, and updated_at attributes, as well as the _start method that is called by the downloader to start the resource download.

FileResource (abstract) provides base features for writing target files, but does not define _write_target_file method itself.

Features provided by file resources are documented on this page. Concrete classes are documented in dedicated pages.

BaseResources

BaseResource instance have:

• an id (ResourceId): unique identifier for the resource that matches the pattern [\w@-]+,

• dataset_ids (frozenset[str] | None): optional set of dataset IDs this resource belongs to (default None = global resource). See Dataset IDs for details.

• updated_at (datetime | None): optional timestamp indicating when the resource data was last updated on the provider side, allowing incremental mode.

FileResources features

Download flow

The FileResource._start method first calls the FileResource._write_target_file (abstract) method, responsible for writing the target file. It is up to the concrete classes to implement the _write_target_file method.

If the file could not be written, a ResourceFileNotWritten exception is raised (this does not apply to optional resources). Otherwise, after the file is written, the FileResource class handles post-processing features like MIME type validation and file reformatting via dedicated methods.

Optional flag

When a resource is marked as optional=True, the downloader will not consider it as a failure if the target file was not written after the download attempt.

Without optional, a _write_target_file that completes without an exception but without writing the file raises a ResourceFileNotWritten error, incrementing the fail count of the downloader. With optional=True, that else branch is skipped and the resource is silently ignored.

This is useful when a custom subclass overrides _write_target_file and deliberately chooses not to write the file based on business logic (e.g., empty response that shouldn’t be treated as a failure).

Keep flag

By default keep=True, meaning the downloaded file is placed in the source-data directory after a successful download.

When keep=False, the file stays in the .cache/ directory and is not copied to source-data/. This is useful for intermediate files that only need to be cached temporarily for resume mode, but don’t need to be kept in the final output.

Example: ZIP file used as a temporary checkpoint

Some providers distribute data as ZIP archives containing actual data files (e.g., XML or CSV files). Once downloaded, the data files are extracted and the ZIP archive itself is no longer needed in the output.

Using keep=False, the ZIP file stays only in the cache directory for resume mode. The _iter_resources method first yields the ZIP resource, then after the ZIP is downloaded, reads it from the cache and yields each extracted file as a proper HttpResource — benefiting from MIME type validation, auto-reformatting, debug directories, and cache/resume mode:

import zipfile
from collections.abc import Generator
from typing import override


class AbcDownloader(BaseDownloader):
    # [...]

    @override
    def _iter_resources(self) -> Generator[BaseResource, None, None]:
        # 1. Download the ZIP checkpoint (stays in .cache only)
        zip_resource = HttpResource(
            id="dataset1_zip",
            request="https://provider.org/data/dataset1.zip",
            target_file="dataset1.zip",
            keep=False,
        )
        with self._propagate_exception():
            try:
                yield zip_resource
            except ResourceError:
                return  # ZIP download failed, nothing to extract
        # 2. Yield each inner file via StaticFileResource
        with zipfile.ZipFile(zip_resource.target_file, "r") as zf:
            for name in zf.namelist():
                yield StaticFileResource(
                    id=name,
                    content=zf.read(name),
                    target_file=name,
                )

Each extracted file is a standard FileResource and goes through the full FileResource lifecycle: MIME type validation, file reformatting, cache and debug directories, resume mode support — nothing needs to be reimplemented.

Retry

By default, FileResource does not configure any retry strategy (its _default_retrying property returns None), so a download is attempted exactly once.

Subclasses like HttpResource override _default_retrying to provide a retry strategy.

The _start method delegates to run_retrying_attempts which will make multiple attempts according to the configured retrying parameter.

You can pass a custom tenacity BaseRetrying instance via the retrying constructor parameter.

MIME type validation

A common pitfall when downloading files is that the server responds something else than the expected response, the most well-known example being the “404 not found” web page.

By default, the FileResource class validates that the actual MIME type of the downloaded file matches the expected one, based on the file name, after it has been downloaded.

For example, a file nameed catalog.json will be expected to have a MIME type of application/json, and a file named data.csv will be expected to have a MIME type of text/csv.

The FileResource._validate_mimetype method calls the validate_mimetype, which makes use of the mimetypes.guess_type function of the Python standard library.

If the MIME type could not be guessed based on the file name, the MimeTypeNotGuessed exception is raised. In that case it is still possible to pass the accept_mimetype kwarg to the constructor of FileResource, which skips guessing the MIME type from the file name.

The actual MIME type of the file is then detected from the file contents by using the python-magic package. If the detected MIME type does not match the expected one, the InvalidMimeType exception is raised. The BaseDownloader considers the resource as failed and logs the error.

MIME type validation can be disabled by passing the validate_mimetype=False kwarg to the constructor of FileResource.

Reformat files

When downloading a text-based file like JSON or XML, the server can send its contents formatted in different ways. For exemple, a JSON file can be responded completely unindented (as a single line), indented with 2 or 4 spaces, etc. The same goes for XML files.

To minimize variations between different versions of the same file, especially when using a version control system like Git, it is advised to reformat the file using settings that don’t vary in time.

By default, the FileResource class reformats the file after it has been downloaded, using a different method based on the file extension. As of now, the JSON (<file>.json) and XML (<file>.xml) formats are supported.

The FileResource._reformat_file method calls the reformat_file function which can rely on external tools to actually reformat the files. If a tool is missing, or the reformatting fails, an exception is raised and the resource is considered as failed by the BaseDownloader

Reformatting can be disabled by passing the reformat_file=False kwarg to the constructor of FileResource.

Dataset IDs

A resource can optionally declare which dataset(s) it belongs to via the dataset_ids parameter (type set[str] | None).

• None or not set — the resource is global (e.g., a catalog file that applies to all datasets of a provider). It is never filtered by dataset selection options.

• {"GDP"} — the resource belongs to the GDP dataset. It is subject to --datasets, --exclude-datasets and --skip-datasets filtering.

• {"GDP", "INFLATION"} — a single resource that belongs to multiple datasets (e.g., an Excel file containing several datasets). The resource is downloaded if any of its dataset IDs is selected, and skipped only when all its IDs are excluded.

Example:

# A global resource — never filtered by dataset selection
yield HttpResource(
    id="catalog",
    request="https://provider.org/data/catalog.json",
    target_file="catalog.json",
)

# A single-dataset resource — affected by --datasets, --exclude-datasets and --skip-datasets
yield HttpResource(
    id="gdp",
    request="https://provider.org/data/gdp.csv",
    target_file="gdp.csv",
    dataset_ids={"GDP"},
)

# A multi-dataset resource — downloaded if any dataset is selected
yield HttpResource(
    id="excel",
    request="https://provider.org/data/all.xlsx",
    target_file="all.xlsx",
    dataset_ids={"GDP", "INFLATION", "UNEMPLOYMENT"},
)