Resource groups

Resource groups represent files that must be downloaded together, or none if any of them fail.

For example, a dataset may be composed of a data file and a structure file. If the data file is not available, then the structure file is useless, and vice versa.

Lifecycle

  1. _start calls _download, which iterates over child resources and downloads them one by one.

  2. If all child resources succeed, _move_files_to_target_dir moves the files from the cache directory to the target directory atomically (using move_files_transactionally).

  3. If any child resource fails, the target directory is left untouched and the failed resources’ files are moved to the debug directory.

Resource groups by default store the files of the child resources in the source-data directory, but if the target_dir kwarg is passed to its constructor, they will be stored under that base directory. The item_kind parameter can be set to customize the log messages (e.g., "page", "partition").

Example: data and structure files of a dataset

Let’s download a dataset composed of 2 files: data.xml and structure.xml.

import responses

class MyDownloader(BaseDownloader):
    # [...]

    @override
    def start(self) -> None:
        with responses.RequestsMock(assert_all_requests_are_fired=False) as rsps:
            rsps.add(
                responses.GET,
                "https://abc-provider.com/dataset1/data.xml",
                body="<p>Page Not Found<p>",
                content_type="text/html",
                status=404,
            )
            rsps.add(
                responses.GET,
                "https://abc-provider.com/dataset1/structure.xml",
                body='<?xml version="1.0" encoding="UTF-8"?><structure />',
                content_type="application/xml",
            )
            super().start()

    @override
    def _iter_resources(self) -> Iterator[BaseResource]:
        yield ResourceGroup(
            id="dataset1",
            resources=[
                HttpResource(
                    id="data",
                    request="https://abc-provider.com/dataset1/data.xml",
                    target_file="data.xml",
                ),
                HttpResource(
                    id="structure",
                    request="https://abc-provider.com/dataset1/structure.xml",
                    target_file="structure.xml",
                ),
            ],
            target_dir="dataset1",
        )

The source-data/dataset1 directory does not exist, which is what we want: one of the resources of the group failed, so we want none of them.

The structure.xml file is stored in the cache directory so that a subsequent download will take advantage of the resume mode to skip downloading the file again.

The response dump is stored in the debug directory as data.xml.http_dump.attempt_1.txt and allows us to inspect what’s going on.

Let’s fix the simulated response to make the data resource succeed:

src/abc_fetcher/downloader.py
import responses

class AbcDownloader(BaseDownloader):
    # [...]

    @override
    def start(self) -> None:
        with responses.RequestsMock(assert_all_requests_are_fired=False) as rsps:
            rsps.add(
                responses.GET,
                "https://abc-provider.com/dataset1/data.xml",
                body='<?xml version="1.0" encoding="UTF-8"?><data />',
                content_type="application/xml",
            )
            rsps.add(
                responses.GET,
                "https://abc-provider.com/dataset1/structure.xml",
                body='<?xml version="1.0" encoding="UTF-8"?><structure />',
                content_type="application/xml",
            )
            super().start()

Let’s run the script again:

python download.py source-data

Let’s look at the source-data directory:

$ tree -a source-data
source-data
├── .cache
│   └── .gitignore
├── dataset1
│   ├── data.xml
│   └── structure.xml
└── .debug
    └── .gitignore

Now the source-data/dataset1 directory exists, and contains all the files of the resource group.

Other kinds of groups

For downloading paginated APIs or partitioned datasets, see the dedicated pages: