What is a fetcher

Read the fetcher page of the DBnomics documentation for a general introduction of fetchers.

Global fetcher architecture

The following diagram unifies both the download and convert processes into a single overview of a fetcher:

        flowchart TB
    AbcSourceDataDirManager
    source_data@{ shape: cyl, label: "Source data" }
    converted_data@{ shape: cyl, label: "Converted data" }

    download@{ shape: lean-r, label: "download.py" }
    provider_infra@{ shape: cloud, label: "Provider infrastructure" }

    download -->|call .start| AbcDownloader
    HttpResource -->|fetch| provider_infra
    AbcDownloader -->|create| HttpResource
    AbcDownloader --> AbcSourceDataDirManager
    HttpResource -->|write| source_data

    convert@{ shape: lean-r, label: "convert.py" }

    convert -->|call .start| AbcConverter
    AbcConverter -->|create| MyDatasetConverter
    MyDatasetConverter --> AbcSourceDataDirManager
    AbcSourceDataDirManager -->|read| source_data
    MyDatasetConverter -->|write| converted_data
    

Design principles

Be standalone programs

Fetchers can run independently from any infrastructure: they just write data to the file-system.

This allows anyone to run them without having to run the complete DBnomics infrastructure.

However, when fetchers are run by the DBnomics infrastructure, the output is stored in the DBnomics data repository and made available on DBnomics website and web API.

2 stages: download and convert

A fetcher is a 2 stage process: download and convert.

First it downloads data from the provider and store it in its original format (called source data).

File formats can be more or less structured: XML, JSON, CSV, XLSX, ad-hoc text files, etc. and distributed through static files (sometimes called bulk download) or web APIs.

Then the fetcher converts source data into DBnomics data model and store it in the converted data directory.

Aim for maximum data coverage

Fetchers should handle a maximum of data from their corresponding provider.

For example, if a provider ships source data through its web API in SDMX format, the fetcher should cover all the datasets by iterating the list of datasets dynamically. As data is structured, processing one dataset costs the same than processing 10000 datasets.

That’s not always easy: there are sometimes particular cases to handle, and sometimes providers ship manually-formatted Excel files that require handling them separately. In those cases, the fetcher author can decide to cover a subset of the available datasets.

Run frequently and keep snapshots

Fetchers are designed to be run every day (or more).

As a consequence, all the data should not be downloaded everytime the fetcher runs, especially if data is huge.

To help targeting only the datasets that changed since the last fetcher execution, we can rely on release dates when providers make them available.

The DBnomics data repository is designed to store each snapshot of data, for source data and converted data. This allows users to access historical data and ensures reproducibility of analyses, even if most of the time providers do not give access to past versions of data.

This is not something that fetchers have to care about: the DBnomics infrastructure takes care of storing each snapshot. Fetchers only have to download and convert data from the provider, and this output will be considered as the latest snapshot of the provider data.

Downloaded data sometimes differs slightly from one download to another, even if both downloads correspond to the same snapshot.

For example, there can be a prepared_at date in an XML file, or a random URL to a CSS stylesheet in an HTML file used to bypass the browser cache.

Keeping them would create false snapshots, so fetchers should remove those specificities in downloaded data in order to avoid them.

In the same spirit, source data should be reformatted in a standard way.

Handle errors gracefully

Errors may occur during the download or conversion stages.

Errors should not break the entire script execution. Instead, the error should be logged and the next item should start being processed.

For example, a download script may fail downloading a resource because the server is down, or a convert script may fail converting a dataset because data is different than expected.

This default behavior can be modified by using script options like --fail-fast, which makes the script fail by raising an exception as soon as an error is encountered.