Downloading data¶

Overview¶

Here is the big picture of the different components that allow for data download (the data conversion components are grayed out):

        flowchart TB
    AbcSourceDataDirManager
    source_data@{ shape: cyl, label: "Source data" }
    converted_data@{ shape: cyl, label: "Converted data" }

    download@{ shape: lean-r, label: "download.py" }
    provider_infra@{ shape: cloud, label: "Provider infrastructure" }

    download -->|call .start| AbcDownloader
    HttpResource -->|fetch| provider_infra
    AbcDownloader -->|create| HttpResource
    AbcDownloader --> AbcSourceDataDirManager
    HttpResource -->|write| source_data

    convert@{ shape: lean-r, label: "convert.py" }

    convert -.->|call .start| AbcConverter
    AbcConverter -.->|create| MyDatasetConverter
    MyDatasetConverter -.-> AbcSourceDataDirManager
    AbcSourceDataDirManager -->|read| source_data
    MyDatasetConverter -.->|write| converted_data

    classDef muted fill:#D3D3D31F,stroke:lightgray,stroke-dasharray: 5 5;
    class convert,AbcConverter,MyDatasetConverter,converted_data muted;

Download script¶

This script can be called manually from the command line, and will be called by the DBnomics infrastructure in production.

Note the separation of concerns:

the DownloadCLI class handles the CLI arguments and options, then creates and runs the downloader,
the AbcDownloader class takes care of the download logic.

Downloader¶

The AbcDownloader class, inherited from BaseDownloader, is responsible for downloading data from the infrastructure of this specific provider.

To achieve its goal, it creates file resources and downloads them. In the following file, a first resource is created for the catalog.json file that lists the datasets made available by the provider, then subsequent resources are created for each dataset file, based on the content of the catalog.

Note

As we use a dummy provider, we use responses to mock the dummy URLs that start with https://abc-provider.com/ in the AbcDownloader.start method.

In a regular fetcher, you won’t have to do this: just remove the start method.

The URL for catalog.json is declared twice to simulate a busy server. We’ll see in the next sections how the downloader can handle this kind of error and retry the download.

The _iter_resources method yields resources that will be downloaded by the BaseDownloader. Each downloader must implement this abstract method. See the Resources section for more details.

The _source_data_dir_manager cached property instantiates the AbcSourceDataDirManager class lazily, which is a fundamental pattern detailed in Source data directory manager section.

Resources¶

A file resource represents some data file made available by the provider that will be stored as a single file. It encapsulates the logic to download that file and post-process it.

The HttpResource class allows downloading a file from a HTTP/HTTPS URL using requests. It offers convenient features such as validating the MIME type of the target file, reformatting its content according to its format (e.g. XML, JSON, etc.), and provides several customization parameters and extension hooks. Those features are enabled by default with auto-detection, but can be disabled by passing arguments to the constructor.

Resource groups download several resource all together, or none if any of them fail.

To learn more about resources, read the Resources section.

Source data directory manager¶

We introduce a AbcSourceDataDirManager class to encapsulate the logic for accessing and parsing source data files, and use it from both the AbcDownloader and the AbcConverter.

As source data files are written by the downloader and read back by the converter, we don’t want to duplicate this logic.

In our example case, the ABC provider exposes its datasets in a catalog.json file such as:

[
  {"dataset_id": "GAS_PRICE", "dataset_name": "Gas price", "updated_at": "2025-01-15"},
]

This catalog will be used by the downloader to iterate over the datasets to download them sequentially, and maybe later by the converter to produce a category tree. So we want to model this catalog and make it accessible as a iter_catalog_items method of the AbcSourceDataDirManager class.

First, those catalog items can be modeled by a CatalogItem dataclass in a separate module:

The iter_catalog_items method is called by the AbcDownloader in order to download all the datasets.

To parse and validate the actual catalog.json file, we use a TypedCodec from dbnomics-toolbox, which is a convenient way to decode and validate data files with a given data type.

Run the download script¶

Note

TOOLBOX_NO_LOG_COLOR=1 disables colors in the logs, TOOLBOX_NO_LOG_TIMESTAMP=1 hides timestamps in the logs, and TOOLBOX_PLAIN_TRACEBACK=1 shows plain exception tracebacks, for the sake of readability in the documentation.

$ rm -rf source-data

$ TOOLBOX_NO_LOG_COLOR=1 TOOLBOX_NO_LOG_TIMESTAMP=1 TOOLBOX_PLAIN_TRACEBACK=1 uv run download.py source-data
[debug    ] Created target directory: 'source-data' [dbnomics_toolbox.fetcher_utils.processors.base_downloader]
[debug    ] Created cache directory: 'source-data/.cache' [dbnomics_toolbox.fetcher_utils.processors.base_downloader]
[debug    ] Created debug directory: 'source-data/.debug' [dbnomics_toolbox.fetcher_utils.processors.base_downloader]
[debug    ] Start downloading resource HttpResource(id='catalog') [dbnomics_toolbox.fetcher_utils.processors.base_downloader]
[debug    ] Fetching URL 'https://abc-provider.com/data/catalog.json' (connect timeout: 1 minute, read timeout: 1 minute)... [dbnomics_toolbox.fetcher_utils.resources.http_resource]
[debug    ] Invalid HTTP response: <Response [429]> [dbnomics_toolbox.fetcher_utils.resources.http_resource]
[debug    ] Dumped HTTP request and response to 'source-data/.debug/catalog.json.http_dump.attempt_1.txt' (321 bytes) [dbnomics_toolbox.fetcher_utils.resources.http_resource]
[error    ] Error during attempt 1 after 0 seconds [dbnomics_toolbox.retry_utils.loggers]
Traceback (most recent call last):
  File "/home/cbenz/Dev/dbnomics/dbnomics-toolbox/src/dbnomics_toolbox/retry_utils/loggers.py", line 42, in log_failed_attempt
    outcome.result()
    ~~~~~~~~~~~~~~^^
  File "/home/cbenz/.local/share/uv/python/cpython-3.13.12-linux-x86_64-gnu/lib/python3.13/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "/home/cbenz/.local/share/uv/python/cpython-3.13.12-linux-x86_64-gnu/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/cbenz/Dev/dbnomics/dbnomics-toolbox/src/dbnomics_toolbox/retry_utils/run.py", line 30, in run_retrying_attempts
    result = run_attempt(retry_state=retry_state)
  File "/home/cbenz/Dev/dbnomics/dbnomics-toolbox/src/dbnomics_toolbox/fetcher_utils/resources/file_resource.py", line 224, in run_attempt
    self._write_target_file(retry_state=retry_state)
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cbenz/Dev/dbnomics/dbnomics-toolbox/src/dbnomics_toolbox/fetcher_utils/resources/http_resource.py", line 235, in _write_target_file
    with self._fetch_response(retry_state=retry_state) as response:
         ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cbenz/.local/share/uv/python/cpython-3.13.12-linux-x86_64-gnu/lib/python3.13/contextlib.py", line 141, in __enter__
    return next(self.gen)
  File "/home/cbenz/Dev/dbnomics/dbnomics-toolbox/src/dbnomics_toolbox/fetcher_utils/resources/http_resource.py", line 165, in _fetch_response
    self._validate_response(response)
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/home/cbenz/Dev/dbnomics/dbnomics-toolbox/src/dbnomics_toolbox/fetcher_utils/resources/http_resource.py", line 229, in _validate_response
    response.raise_for_status()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/cbenz/Dev/dbnomics/dbnomics-fetchers/abc-fetcher/.venv/lib/python3.13/site-packages/requests/models.py", line 1028, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://abc-provider.com/data/catalog.json
[debug    ] Sleeping 1.5 seconds           [dbnomics_toolbox.retry_utils.loggers]
[debug    ] Starting attempt 2             [dbnomics_toolbox.retry_utils.loggers]
[debug    ] Fetching URL 'https://abc-provider.com/data/catalog.json' (connect timeout: 1 minute, read timeout: 1 minute)... [dbnomics_toolbox.fetcher_utils.resources.http_resource]
[debug    ] Received HTTP response: <Response [200]> [dbnomics_toolbox.fetcher_utils.resources.http_resource]
[debug    ] Started writing to temporary file 'source-data/catalog.part' (0 bytes)... [dbnomics_toolbox.fetcher_utils.file_utils.common]
[debug    ] Wrote file 'source-data/catalog.json' (179 bytes) [dbnomics_toolbox.fetcher_utils.file_utils.common] duration='0 seconds'
[debug    ] Start reformatting JSON file 'source-data/catalog.json' (179 bytes) with command ['/usr/bin/jq', '--indent', '2'] [dbnomics_toolbox.fetcher_utils.file_utils.json_utils]
[debug    ] Reformatted JSON file 'source-data/catalog.json' (179 bytes) [dbnomics_toolbox.fetcher_utils.file_utils.common] duration='0 seconds' initial_file_size='179 bytes'
[info     ] Finished downloading resource 'catalog' successfully [dbnomics_toolbox.fetcher_utils.processors.base_downloader] duration='1.52 seconds'
[debug    ] Start downloading resource HttpResource(id='GAS_PRICE') [dbnomics_toolbox.fetcher_utils.processors.base_downloader]
[debug    ] Fetching URL 'https://abc-provider.com/data/GAS_PRICE.csv' (connect timeout: 1 minute, read timeout: 1 minute)... [dbnomics_toolbox.fetcher_utils.resources.http_resource]
[debug    ] Received HTTP response: <Response [200]> [dbnomics_toolbox.fetcher_utils.resources.http_resource]
[debug    ] Started writing to temporary file 'source-data/datasets/GAS_PRICE/data.part' (0 bytes)... [dbnomics_toolbox.fetcher_utils.file_utils.common]
[debug    ] Wrote file 'source-data/datasets/GAS_PRICE/data.csv' (205 bytes) [dbnomics_toolbox.fetcher_utils.file_utils.common] duration='0 seconds'
[debug    ] Set updated_at=2025-01-15T00:00:00+00:00 for dataset 'GAS_PRICE' [dbnomics_toolbox.fetcher_utils.processors.downloader_state_repo]
[info     ] Finished downloading resource 'GAS_PRICE' successfully [dbnomics_toolbox.fetcher_utils.processors.base_downloader] duration='0 seconds'
[debug    ] Start downloading resource HttpResource(id='GDP') [dbnomics_toolbox.fetcher_utils.processors.base_downloader]
[debug    ] Fetching URL 'https://abc-provider.com/data/GDP.csv' (connect timeout: 1 minute, read timeout: 1 minute)... [dbnomics_toolbox.fetcher_utils.resources.http_resource]
[debug    ] Invalid HTTP response: <Response [404]> [dbnomics_toolbox.fetcher_utils.resources.http_resource]
[debug    ] Dumped HTTP request and response to 'source-data/.debug/datasets/GDP/data.csv.http_dump.attempt_1.txt' (293 bytes) [dbnomics_toolbox.fetcher_utils.resources.http_resource]
[error    ] Error downloading resource 'GDP' [dbnomics_toolbox.fetcher_utils.processors.base_downloader] duration='0 seconds'
Traceback (most recent call last):
  File "/home/cbenz/Dev/dbnomics/dbnomics-toolbox/src/dbnomics_toolbox/fetcher_utils/processors/base_downloader.py", line 145, in _download_resource
    resource._start()  # type: ignore[reportPrivateUsage] # noqa: SLF001
    ~~~~~~~~~~~~~~~^^
  File "/home/cbenz/Dev/dbnomics/dbnomics-toolbox/src/dbnomics_toolbox/fetcher_utils/resources/file_resource.py", line 227, in _start
    run_retrying_attempts(
    ~~~~~~~~~~~~~~~~~~~~~^
        retrying=self._retrying,
        ^^^^^^^^^^^^^^^^^^^^^^^^
        run_attempt=run_attempt,
        ^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/cbenz/Dev/dbnomics/dbnomics-toolbox/src/dbnomics_toolbox/retry_utils/run.py", line 26, in run_retrying_attempts
    for attempt in retrying:
                   ^^^^^^^^
  File "/home/cbenz/Dev/dbnomics/dbnomics-fetchers/abc-fetcher/.venv/lib/python3.13/site-packages/tenacity/__init__.py", line 438, in __iter__
    do = self.iter(retry_state=retry_state)
  File "/home/cbenz/Dev/dbnomics/dbnomics-fetchers/abc-fetcher/.venv/lib/python3.13/site-packages/tenacity/__init__.py", line 371, in iter
    result = action(retry_state)
  File "/home/cbenz/Dev/dbnomics/dbnomics-fetchers/abc-fetcher/.venv/lib/python3.13/site-packages/tenacity/__init__.py", line 393, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
                                     ~~~~~~~~~~~~~~~~~^^
  File "/home/cbenz/.local/share/uv/python/cpython-3.13.12-linux-x86_64-gnu/lib/python3.13/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "/home/cbenz/.local/share/uv/python/cpython-3.13.12-linux-x86_64-gnu/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/cbenz/Dev/dbnomics/dbnomics-toolbox/src/dbnomics_toolbox/retry_utils/run.py", line 30, in run_retrying_attempts
    result = run_attempt(retry_state=retry_state)
  File "/home/cbenz/Dev/dbnomics/dbnomics-toolbox/src/dbnomics_toolbox/fetcher_utils/resources/file_resource.py", line 224, in run_attempt
    self._write_target_file(retry_state=retry_state)
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cbenz/Dev/dbnomics/dbnomics-toolbox/src/dbnomics_toolbox/fetcher_utils/resources/http_resource.py", line 235, in _write_target_file
    with self._fetch_response(retry_state=retry_state) as response:
         ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cbenz/.local/share/uv/python/cpython-3.13.12-linux-x86_64-gnu/lib/python3.13/contextlib.py", line 141, in __enter__
    return next(self.gen)
  File "/home/cbenz/Dev/dbnomics/dbnomics-toolbox/src/dbnomics_toolbox/fetcher_utils/resources/http_resource.py", line 165, in _fetch_response
    self._validate_response(response)
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/home/cbenz/Dev/dbnomics/dbnomics-toolbox/src/dbnomics_toolbox/fetcher_utils/resources/http_resource.py", line 229, in _validate_response
    response.raise_for_status()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/cbenz/Dev/dbnomics/dbnomics-fetchers/abc-fetcher/.venv/lib/python3.13/site-packages/requests/models.py", line 1028, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://abc-provider.com/data/GDP.csv
[info     ] The report has been saved to 'download_report.json' (85 bytes) [dbnomics_toolbox.fetcher_utils.processors.base_processor]
[info     ] 3 resources processed: 2 succeeded, 1 failed and 0 skipped. [dbnomics_toolbox.fetcher_utils.processors.base_downloader]
[debug    ] Downloader state not saved because no output state file was provided by the user, logging state data instead: DownloaderState(dataset_updates={'GAS_PRICE': datetime.datetime(2025, 1, 15, 0, 0, tzinfo=datetime.timezone.utc)}) [dbnomics_toolbox.fetcher_utils.cli_utils.download_cli]

The log of the script shows that the resource catalog was downloaded successfully, after failing because of a “Server is busy” error, and was reformatted. The retry was automatically handled by the downloader.

The GAS_PRICE dataset was downloaded successfully, but the GDP dataset failed to be downloaded because of a 404 “Page not found” error, which is expected as we simulated that in the AbcDownloader.start method. Please note that 404 HTTP errors do not trigger a retry automatically.

Let’s have a look at the downloaded files:

$ tree -a source-data
source-data
├── .cache
│   └── .gitignore
├── catalog.json
├── datasets
│   ├── GAS_PRICE
│   │   └── data.csv
│   └── GDP
└── .debug
    ├── catalog.json.http_dump.attempt_1.txt
    ├── datasets
    │   └── GDP
    │       └── data.csv.http_dump.attempt_1.txt
    └── .gitignore

8 directories, 6 files

JSON files are reformatted automatically:

$ cat source-data/catalog.json
[
  {
    "dataset_id": "GAS_PRICE",
    "dataset_name": "Gas price",
    "updated_at": "2025-01-15"
  },
  {
    "dataset_id": "GDP",
    "dataset_name": "Gross domestic product",
    "updated_at": "2025-01-10"
  }
]

The .cache directory is not used in this case. It is where the downloader stores files that are not yet ready to be moved to the source-data directory, for example because they are still being downloaded, or because they failed post-download validation, to prevent downloading them several times from the provider infrastructure, to spare bandwidth and speed up the download process.

The .debug directory is used to store debug files associated to resources that failed downloading, for example in the case of HTTP 4xx or 5xx errors. The following section Error handling covers this topic.

Read the Base classes page to learn more about these features.

Resume mode¶

When the file of a resource already exists in source-data, that resource is skipped. This behavior is called the resume mode.

For example, if we run the download script again, the logs show that the resource is skipped:

$ TOOLBOX_NO_LOG_COLOR=1 TOOLBOX_NO_LOG_TIMESTAMP=1 TOOLBOX_PLAIN_TRACEBACK=1 uv run download.py source-data
[...]
[debug    ] Skipped resource HttpResource(id='catalog'): [Resume mode] Skipping resource 'catalog' because its file already exists: 'source-data/catalog.json' (218 bytes) [dbnomics_toolbox.fetcher_utils.processors.base_downloader]
[debug    ] Skipped resource HttpResource(id='GAS_PRICE'): [Resume mode] Skipping resource 'GAS_PRICE' because its file already exists: 'source-data/datasets/GAS_PRICE/data.csv' (205 bytes) [dbnomics_toolbox.fetcher_utils.processors.base_downloader]
[...]

The resume mode can be disabled by passing the --no-resume option: all the resources will be downloaded, whether their file already exists or not.

To re-download a particular resource file only while keeping the resume mode enabled, just delete its file and re-execute the script.

Incremental mode¶

Use incremental mode to avoid downloading unchanged data.

When a resource has an updated_at date, the downloader persists it in its state. On subsequent runs, if the persisted updated_at matches the current one, the resource is skipped — the data hasn’t changed.

In practice, this update date can be available at different places: in the catalog, in a calendar RSS stream, on the website in HTML, or in the resource file itself.

When no update date is available for a particular resource, the downloader will always download this resource.

The download CLI provides options to control incremental mode:

--incremental/--no-incremental: Enable or disable incremental mode (default: enabled, TOOLBOX_INCREMENTAL). When disabled, all dataset resources are downloaded regardless of their updated_at.
--input-state-file (TOOLBOX_INPUT_STATE_FILE): Path to a JSON file produced by a previous download run (via --output-state-file). Contains the last known updated_at for each dataset. If a dataset’s current updated_at matches the one in this file, it is skipped.
--output-state-file (TOOLBOX_OUTPUT_STATE_FILE): Path where the downloader writes the state after a successful run. This file can be re-injected on the next run via --input-state-file.
--updated-after (TOOLBOX_UPDATED_AFTER): An arbitrary cutoff date (ISO format). Only datasets with an updated_at strictly after this date are downloaded.

On the first run, write the state to a file:

$ rm -rf downloader_state.json source-data

$ TOOLBOX_NO_LOG_COLOR=1 TOOLBOX_NO_LOG_TIMESTAMP=1 TOOLBOX_PLAIN_TRACEBACK=1 uv run download.py --output-state-file downloader_state.json source-data

The output is not shown because it is almost identical to the previous run, except that the downloader state is written to downloader_state.json:

Note

The output state file does not contain any entry for the GDP dataset, even if its update date was known in the catalog.

This is because the download of GDP failed (with a 404 HTTP error) so we can’t consider it as downloaded successfully, and we don’t want to skip it on the next run.

On subsequent runs, re-inject the state to skip unchanged datasets:

$ TOOLBOX_NO_LOG_COLOR=1 TOOLBOX_NO_LOG_TIMESTAMP=1 TOOLBOX_PLAIN_TRACEBACK=1 uv run download.py --input-state-file downloader_state.json --output-state-file downloader_state.json source-data
[...]
[debug    ] Skipped resource HttpResource(id='GAS_PRICE'): [Incremental mode] Skipping 'GAS_PRICE' (updated_at=2025-01-15T00:00:00+00:00) because it has not been updated after the previous updated_at known date: 2025-01-15T00:00:00+00:00 [dbnomics_toolbox.fetcher_utils.processors.base_downloader]
[...]

Error handling¶

If any exception occurs while downloading a resource, the resource will be skipped and the script continue without crashing, and the error will be logged.

If the target file of the resource was written, even partially, the BaseDownloader will move the file to the debug directory for further inspection.

By default, the debug directory is a sub-directory of the source data directory named .debug. Its path can be customized by passing the --debug-dir option.

Debug files prevents the developer to lose the information about what went wrong, and allows to inspect the problem in context without having to reproduce it with curl or in the browser.

In our previous example, the catalog failed a first time because of a “Server busy” error, and the GDP dataset failed because of a 404 “Page not found” error. We can inspect the corresponding debug files:

Modeling the provider website¶

When the provider has a website, it is a good idea to model it as a Website class that provides high-level methods to access the data of the website, and to encapsulate all the knowledge about the website in one place.

For example, sometimes we need to scrape the website of a provider, even if it provides an API, because some particular data might only be available on the website.

For example, ons-fetcher relies on web scraping to extract the category tree of datasets in its website.py module.

Fetchers can define a website.py module that exposes a Website class:

This class will be used by the converter in the next section to incorporate the URL of each dataset on the website of the provider in converted data.