# Base classes

## Class diagram

```{mermaid}
classDiagram
    BaseResource <|-- FileResource
    BaseResource <|-- ResourceGroup
    FileResource <|-- HttpResource
    HttpResource <|-- SdmxResource
    ResourceGroup <|-- BasePageResourceGroup
    ResourceGroup <|-- BasePartitionGroup

    class BaseResource {
        <<abstract>>
        +id: ResourceId
        +dataset_ids: frozenset[str] | None
        +updated_at: datetime | None
        +_start()
    }

    class FileResource {
        <<abstract>>
        +target_file: Path
        +keep: bool
        +optional: bool
        +_reformat_file()
        +_validate_mimetype()
        +_write_target_file(retry_state)
    }

    class HttpResource {
        +request: Request | str | URL
        +session: Session | None
        +dump_response: DumpResponse
        +timeout: Timeout | Timeouts | None
    }

    class SdmxResource {
        +normalize_sdmx_file_header: bool
        +raise_for_sdmx_error(error)
    }

    class ResourceGroup {
        +resources: list[FileResource]
        +target_dir: Path | None
        +_move_files_to_target_dir()
        +_write_target_file()
    }

    class BasePageResourceGroup {
        <<abstract>>
        +start_page_num: int
        +_create_resource(page_num) FileResource
    }

    class BasePartitionGroup {
        <<abstract>>
        +initial_partitions: list[TPartition]
        +_create_resource(partition, ...) FileResource
    }
```

[](#BaseResource) (abstract) is the base class for all resources. It defines the `id`, `type`, and `updated_at` attributes, as well as the `_start` method that is called by the downloader to start the resource download.

[](#FileResource) (abstract) provides base features for writing target files, but does not define `_write_target_file` method itself.

Features provided by file resources are documented on this page.
Concrete classes are documented in dedicated pages.

## `BaseResources`

[](#BaseResource) instance have:

- an `id` (`ResourceId`): unique identifier for the resource that matches the pattern `[\w@-]+`,
- `dataset_ids` (`frozenset[str] | None`): optional set of dataset IDs this resource belongs to (default `None` = global resource). See [](#dataset-ids) for details.
- `updated_at` (`datetime | None`): optional timestamp indicating when the resource data was last updated on the provider side, allowing [incremental mode](../tutorial/downloading-data.md#incremental-mode).

## `FileResources` features

## Download flow

The `FileResource._start` method first calls the `FileResource._write_target_file` (abstract) method, responsible for writing the target file. It is up to the concrete classes to implement the `_write_target_file` method.

If the file could not be written, a [](#ResourceFileNotWritten) exception is raised (this does not apply to optional resources).
Otherwise, after the file is written, the `FileResource` class handles post-processing features like MIME type validation and file reformatting via dedicated methods.

## Optional flag

When a resource is marked as `optional=True`, the downloader will not consider it as a failure if the target file was not written after the download attempt.

Without `optional`, a `_write_target_file` that completes without an exception but without writing the file raises a [](#ResourceFileNotWritten) error, incrementing the fail count of the downloader.
With `optional=True`, that `else` branch is skipped and the resource is silently ignored.

This is useful when a custom subclass overrides `_write_target_file` and deliberately chooses not to write the file based on business logic (e.g., empty response that shouldn't be treated as a failure).

## Keep flag

By default `keep=True`, meaning the downloaded file is placed in the `source-data` directory after a successful download.

When `keep=False`, the file stays in the `.cache/` directory and is **not** copied to `source-data/`. This is useful for intermediate files that only need to be cached temporarily for resume mode, but don't need to be kept in the final output.

### Example: ZIP file used as a temporary checkpoint

Some providers distribute data as ZIP archives containing actual data files (e.g., XML or CSV files). Once downloaded, the data files are extracted and the ZIP archive itself is no longer needed in the output.

Using `keep=False`, the ZIP file stays only in the cache directory for resume mode. The `_iter_resources` method first yields the ZIP resource, then after the ZIP is downloaded, reads it from the cache and yields each extracted file as a proper [](#HttpResource) — benefiting from MIME type validation, auto-reformatting, debug directories, and cache/resume mode:

```python
import zipfile
from collections.abc import Generator
from typing import override


class AbcDownloader(BaseDownloader):
    # [...]

    @override
    def _iter_resources(self) -> Generator[BaseResource, None, None]:
        # 1. Download the ZIP checkpoint (stays in .cache only)
        zip_resource = HttpResource(
            id="dataset1_zip",
            request="https://provider.org/data/dataset1.zip",
            target_file="dataset1.zip",
            keep=False,
        )
        with self._propagate_exception():
            try:
                yield zip_resource
            except ResourceError:
                return  # ZIP download failed, nothing to extract
        # 2. Yield each inner file via StaticFileResource
        with zipfile.ZipFile(zip_resource.target_file, "r") as zf:
            for name in zf.namelist():
                yield StaticFileResource(
                    id=name,
                    content=zf.read(name),
                    target_file=name,
                )
```

Each extracted file is a standard `FileResource` and goes through the full `FileResource` lifecycle: MIME type validation, file reformatting, cache and debug directories, resume mode support — nothing needs to be reimplemented.

## Retry

By default, [](#FileResource) does not configure any retry strategy (its `_default_retrying` property returns `None`), so a download is attempted exactly once.

Subclasses like [](#HttpResource) override `_default_retrying` to provide a retry strategy.

The `_start` method delegates to [](#run_retrying_attempts) which will make multiple attempts according to the configured `retrying` parameter.

You can pass a custom [tenacity `BaseRetrying`](https://tenacity.readthedocs.io/en/latest/api.html#tenacity.BaseRetrying) instance via the `retrying` constructor parameter.

## MIME type validation

A common pitfall when downloading files is that the server responds something else than the expected response, the most well-known example being the "404 not found" web page.

By default, the [](#FileResource) class validates that the actual MIME type of the downloaded file matches the expected one, based on the file name, after it has been downloaded.

For example, a file nameed `catalog.json` will be expected to have a MIME type of `application/json`, and a file named `data.csv` will be expected to have a MIME type of `text/csv`.

The `FileResource._validate_mimetype` method calls the [](#validate_mimetype), which makes use of the [`mimetypes.guess_type`](https://docs.python.org/3/library/mimetypes.html#mimetypes.guess_type) function of the Python standard library.

If the MIME type could not be guessed based on the file name, the [](#MimeTypeNotGuessed) exception is raised.
In that case it is still possible to pass the `accept_mimetype` kwarg to the constructor of [](#FileResource), which skips guessing the MIME type from the file name.

The actual MIME type of the file is then detected from the file contents by using the [`python-magic`](https://pypi.org/project/python-magic) package.
If the detected MIME type does not match the expected one, the [](#InvalidMimeType) exception is raised.
The [](#BaseDownloader) considers the resource as failed and logs the error.

MIME type validation can be disabled by passing the `validate_mimetype=False` kwarg to the constructor of [](#FileResource).

## Reformat files

When downloading a text-based file like JSON or XML, the server can send its contents formatted in different ways.
For exemple, a JSON file can be responded completely unindented (as a single line), indented with 2 or 4 spaces, etc.
The same goes for XML files.

To minimize variations between different versions of the same file, especially when using a version control system like [Git](https://git-scm.com/), it is advised to reformat the file using settings that don't vary in time.

By default, the [](#FileResource) class reformats the file after it has been downloaded, using a different method based on the file extension.
As of now, the JSON (`<file>.json`) and XML (`<file>.xml`) formats are supported.

The `FileResource._reformat_file` method calls the [](#reformat_file) function which can rely on external tools to actually reformat the files.
If a tool is missing, or the reformatting fails, an exception is raised and the resource is considered as failed by the [](#BaseDownloader)

Reformatting can be disabled by passing the `reformat_file=False` kwarg to the constructor of [](#FileResource).

## Dataset IDs

A resource can optionally declare which dataset(s) it belongs to via the `dataset_ids` parameter (type `set[str] | None`).

- `None` or not set — the resource is **global** (e.g., a catalog file that applies to all datasets of a provider). It is never filtered by dataset selection options.
- `{"GDP"}` — the resource belongs to the `GDP` dataset. It is subject to `--datasets`, `--exclude-datasets` and `--skip-datasets` filtering.
- `{"GDP", "INFLATION"}` — a single resource that belongs to multiple datasets (e.g., an Excel file containing several datasets). The resource is downloaded if any of its dataset IDs is selected, and skipped only when all its IDs are excluded.

Example:

```python
# A global resource — never filtered by dataset selection
yield HttpResource(
    id="catalog",
    request="https://provider.org/data/catalog.json",
    target_file="catalog.json",
)

# A single-dataset resource — affected by --datasets, --exclude-datasets and --skip-datasets
yield HttpResource(
    id="gdp",
    request="https://provider.org/data/gdp.csv",
    target_file="gdp.csv",
    dataset_ids={"GDP"},
)

# A multi-dataset resource — downloaded if any dataset is selected
yield HttpResource(
    id="excel",
    request="https://provider.org/data/all.xlsx",
    target_file="all.xlsx",
    dataset_ids={"GDP", "INFLATION", "UNEMPLOYMENT"},
)
```