# Downloading data ## Overview Here is the big picture of the different components that allow for data download (the data conversion components are grayed out): ```{mermaid} flowchart TB AbcSourceDataDirManager source_data@{ shape: cyl, label: "Source data" } converted_data@{ shape: cyl, label: "Converted data" } download@{ shape: lean-r, label: "download.py" } provider_infra@{ shape: cloud, label: "Provider infrastructure" } download -->|call .start| AbcDownloader HttpResource -->|fetch| provider_infra AbcDownloader -->|create| HttpResource AbcDownloader --> AbcSourceDataDirManager HttpResource -->|write| source_data convert@{ shape: lean-r, label: "convert.py" } convert -.->|call .start| AbcConverter AbcConverter -.->|create| MyDatasetConverter MyDatasetConverter -.-> AbcSourceDataDirManager AbcSourceDataDirManager -->|read| source_data MyDatasetConverter -.->|write| converted_data classDef muted fill:#D3D3D31F,stroke:lightgray,stroke-dasharray: 5 5; class convert,AbcConverter,MyDatasetConverter,converted_data muted; ``` ## Download script ```{literalinclude} ../../_external/abc-fetcher/download.py :caption: download.py ``` This script can be called manually from the command line, and will be called by the DBnomics infrastructure in production. Note the separation of concerns: - the [](#DownloadCLI) class handles the CLI arguments and options, then creates and runs the downloader, - the `AbcDownloader` class takes care of the download logic. ## Downloader The `AbcDownloader` class, inherited from [](#BaseDownloader), is responsible for downloading data from the infrastructure of this specific provider. To achieve its goal, it creates file resources and downloads them. In the following file, a first resource is created for the `catalog.json` file that lists the datasets made available by the provider, then subsequent resources are created for each dataset file, based on the content of the catalog. ```{note} As we use a dummy provider, we use [responses](https://github.com/getsentry/responses) to mock the dummy URLs that start with `https://abc-provider.com/` in the `AbcDownloader.start` method. In a regular fetcher, you won't have to do this: just remove the `start` method. The URL for `catalog.json` is declared twice to simulate a busy server. We'll see in the next sections how the downloader can handle this kind of error and retry the download. ``` ```{literalinclude} ../../_external/abc-fetcher/src/abc_fetcher/downloader.py :caption: src/abc_fetcher/downloader.py ``` The `_iter_resources` method yields resources that will be downloaded by the `BaseDownloader`. Each downloader must implement this abstract method. See the [](#resources) section for more details. The `_source_data_dir_manager` cached property instantiates the `AbcSourceDataDirManager` class lazily, which is a fundamental pattern detailed in [](#source-data-directory-manager) section. ## Resources A file resource represents some data file made available by the provider that will be stored as a single file. It encapsulates the logic to download that file and post-process it. The [](#HttpResource) class allows downloading a file from a HTTP/HTTPS URL using [`requests`](https://requests.readthedocs.io/en/latest/). It offers convenient features such as validating the MIME type of the target file, reformatting its content according to its format (e.g. XML, JSON, etc.), and provides several customization parameters and extension hooks. Those features are enabled by default with auto-detection, but can be disabled by passing arguments to the constructor. Resource groups download several resource all together, or none if any of them fail. To learn more about resources, read the [](../resources/index.md) section. ## Source data directory manager We introduce a `AbcSourceDataDirManager` class to encapsulate the logic for accessing and parsing source data files, and use it from both the `AbcDownloader` and the `AbcConverter`. As source data files are written by the downloader and read back by the converter, we don't want to duplicate this logic. In our example case, the ABC provider exposes its datasets in a `catalog.json` file such as: ```json [ {"dataset_id": "GAS_PRICE", "dataset_name": "Gas price", "updated_at": "2025-01-15"}, ] ``` This catalog will be used by the downloader to iterate over the datasets to download them sequentially, and maybe later by the converter to produce a category tree. So we want to model this catalog and make it accessible as a `iter_catalog_items` method of the `AbcSourceDataDirManager` class. First, those catalog items can be modeled by a `CatalogItem` dataclass in a separate module: ```{literalinclude} ../../_external/abc-fetcher/src/abc_fetcher/source_data_model.py :caption: src/abc_fetcher/source_data_model.py ``` ```{literalinclude} ../../_external/abc-fetcher/src/abc_fetcher/source_data_dir_manager.py :caption: src/abc_fetcher/source_data_dir_manager.py ``` The `iter_catalog_items` method is called by the `AbcDownloader` in order to download all the datasets. To parse and validate the actual `catalog.json` file, we use a [](#TypedCodec) from `dbnomics-toolbox`, which is a convenient way to decode and validate data files with a given data type. ## Run the download script ```{note} `TOOLBOX_NO_LOG_COLOR=1` disables colors in the logs, `TOOLBOX_NO_LOG_TIMESTAMP=1` hides timestamps in the logs, and `TOOLBOX_PLAIN_TRACEBACK=1` shows plain exception tracebacks, for the sake of readability in the documentation. ``` ```{literalinclude} ./_output/download.txt ``` The log of the script shows that the resource `catalog` was downloaded successfully, after failing because of a "Server is busy" error, and was reformatted. The retry was automatically handled by the downloader. The `GAS_PRICE` dataset was downloaded successfully, but the `GDP` dataset failed to be downloaded because of a 404 "Page not found" error, which is expected as we simulated that in the `AbcDownloader.start` method. Please note that 404 HTTP errors do not trigger a retry automatically. Let's have a look at the downloaded files: ```{literalinclude} ./_output/tree-source-data.txt ``` JSON files are reformatted automatically: ```{literalinclude} ./_output/cat-catalog-json.txt ``` The `.cache` directory is not used in this case. It is where the downloader stores files that are not yet ready to be moved to the `source-data` directory, for example because they are still being downloaded, or because they failed post-download validation, to prevent downloading them several times from the provider infrastructure, to spare bandwidth and speed up the download process. The `.debug` directory is used to store debug files associated to resources that failed downloading, for example in the case of HTTP 4xx or 5xx errors. The following section [](#error-handling) covers this topic. Read the [](../resources/base-classes.md) page to learn more about these features. ## Resume mode When the file of a resource already exists in `source-data`, that resource is skipped. This behavior is called the *resume mode*. For example, if we run the download script again, the logs show that the resource is skipped: ```{literalinclude} ./_output/download_resume.txt ``` The resume mode can be disabled by passing the `--no-resume` option: all the resources will be downloaded, whether their file already exists or not. To re-download a particular resource file only while keeping the resume mode enabled, just delete its file and re-execute the script. ## Incremental mode Use incremental mode to avoid downloading unchanged data. When a resource has an `updated_at` date, the downloader persists it in its state. On subsequent runs, if the persisted `updated_at` matches the current one, the resource is skipped — the data hasn't changed. In practice, this update date can be available at different places: in the catalog, in a calendar RSS stream, on the website in HTML, or in the resource file itself. When no update date is available for a particular resource, the downloader will always download this resource. The download CLI provides options to control incremental mode: - `--incremental`/`--no-incremental`: Enable or disable incremental mode (default: enabled, `TOOLBOX_INCREMENTAL`). When disabled, all dataset resources are downloaded regardless of their `updated_at`. - `--input-state-file` (`TOOLBOX_INPUT_STATE_FILE`): Path to a JSON file produced by a previous download run (via `--output-state-file`). Contains the last known `updated_at` for each dataset. If a dataset's current `updated_at` matches the one in this file, it is skipped. - `--output-state-file` (`TOOLBOX_OUTPUT_STATE_FILE`): Path where the downloader writes the state after a successful run. This file can be re-injected on the next run via `--input-state-file`. - `--updated-after` (`TOOLBOX_UPDATED_AFTER`): An arbitrary cutoff date (ISO format). Only datasets with an `updated_at` strictly after this date are downloaded. On the first run, write the state to a file: ```{literalinclude} ./_output/download_incremental_run1.txt ``` The output is not shown because it is almost identical to the previous run, except that the downloader state is written to `downloader_state.json`: ```{literalinclude} ../../_external/abc-fetcher/downloader_state.json ``` ```{note} The output state file does not contain any entry for the `GDP` dataset, even if its update date was known in the catalog. This is because the download of `GDP` failed (with a 404 HTTP error) so we can't consider it as downloaded successfully, and we don't want to skip it on the next run. ``` On subsequent runs, re-inject the state to skip unchanged datasets: ```{literalinclude} ./_output/download_incremental_run2.txt ``` ## Error handling If any exception occurs while downloading a resource, the resource will be skipped and the script continue without crashing, and the error will be logged. If the target file of the resource was written, even partially, the [](#BaseDownloader) will move the file to the debug directory for further inspection. By default, the debug directory is a sub-directory of the source data directory named `.debug`. Its path can be customized by passing the `--debug-dir` option. Debug files prevents the developer to lose the information about what went wrong, and allows to inspect the problem in context without having to reproduce it with curl or in the browser. In our previous example, the `catalog` failed a first time because of a "Server busy" error, and the `GDP` dataset failed because of a 404 "Page not found" error. We can inspect the corresponding debug files: ```{literalinclude} ../../_external/abc-fetcher/source-data/.debug/datasets/GDP/data.csv.http_dump.attempt_1.txt ``` ```{literalinclude} ../../_external/abc-fetcher/source-data/.debug/catalog.json.http_dump.attempt_1.txt ``` ## Modeling the provider website When the provider has a website, it is a good idea to model it as a `Website` class that provides high-level methods to access the data of the website, and to encapsulate all the knowledge about the website in one place. For example, sometimes we need to scrape the website of a provider, even if it provides an API, because some particular data might only be available on the website. For example, [ons-fetcher](https://git.nomics.world/dbnomics-fetchers/ons-fetcher/) relies on web scraping to extract the category tree of datasets in its [website.py](https://git.nomics.world/dbnomics-fetchers/ons-fetcher/-/blob/master/src/ons_fetcher/website.py) module. Fetchers can define a `website.py` module that exposes a `Website` class: ```{literalinclude} ../../_external/abc-fetcher/src/abc_fetcher/website.py :caption: src/abc_fetcher/website.py ``` This class will be used by the converter in the next section to incorporate the URL of each dataset on the website of the provider in converted data.