# Resource groups

Resource groups represent files that must be downloaded together, or none if any of them fail.

For example, a dataset may be composed of a data file and a structure file. If the data file is not available, then the structure file is useless, and vice versa.

## Lifecycle

1. `_start` calls `_download`, which iterates over child resources and downloads them one by one.
2. If all child resources succeed, `_move_files_to_target_dir` moves the files from the cache directory to the target directory atomically (using [](#move_files_transactionally)).
3. If any child resource fails, the target directory is left untouched and the failed resources' files are moved to the debug directory.

Resource groups by default store the files of the child resources in the `source-data` directory, but if the `target_dir` kwarg is passed to its constructor, they will be stored under that base directory.
The `item_kind` parameter can be set to customize the log messages (e.g., `"page"`, `"partition"`).

## Example: data and structure files of a dataset

Let's download a dataset composed of 2 files: `data.xml` and `structure.xml`.

```{code-block} python
import responses

class MyDownloader(BaseDownloader):
    # [...]

    @override
    def start(self) -> None:
        with responses.RequestsMock(assert_all_requests_are_fired=False) as rsps:
            rsps.add(
                responses.GET,
                "https://abc-provider.com/dataset1/data.xml",
                body="<p>Page Not Found<p>",
                content_type="text/html",
                status=404,
            )
            rsps.add(
                responses.GET,
                "https://abc-provider.com/dataset1/structure.xml",
                body='<?xml version="1.0" encoding="UTF-8"?><structure />',
                content_type="application/xml",
            )
            super().start()

    @override
    def _iter_resources(self) -> Iterator[BaseResource]:
        yield ResourceGroup(
            id="dataset1",
            resources=[
                HttpResource(
                    id="data",
                    request="https://abc-provider.com/dataset1/data.xml",
                    target_file="data.xml",
                ),
                HttpResource(
                    id="structure",
                    request="https://abc-provider.com/dataset1/structure.xml",
                    target_file="structure.xml",
                ),
            ],
            target_dir="dataset1",
        )
```

The `source-data/dataset1` directory does not exist, which is what we want: one of the resources of the group failed, so we want none of them.

The `structure.xml` file is stored in the cache directory so that a subsequent download will take advantage of the resume mode to skip downloading the file again.

The response dump is stored in the debug directory as `data.xml.http_dump.attempt_1.txt` and allows us to inspect what's going on.

Let's fix the simulated response to make the `data` resource succeed:

```{code-block} python
:caption: src/abc_fetcher/downloader.py
import responses

class AbcDownloader(BaseDownloader):
    # [...]

    @override
    def start(self) -> None:
        with responses.RequestsMock(assert_all_requests_are_fired=False) as rsps:
            rsps.add(
                responses.GET,
                "https://abc-provider.com/dataset1/data.xml",
                body='<?xml version="1.0" encoding="UTF-8"?><data />',
                content_type="application/xml",
            )
            rsps.add(
                responses.GET,
                "https://abc-provider.com/dataset1/structure.xml",
                body='<?xml version="1.0" encoding="UTF-8"?><structure />',
                content_type="application/xml",
            )
            super().start()
```

Let's run the script again:

```bash
python download.py source-data
```

Let's look at the `source-data` directory:

```bash
$ tree -a source-data
source-data
├── .cache
│   └── .gitignore
├── dataset1
│   ├── data.xml
│   └── structure.xml
└── .debug
    └── .gitignore
```

Now the `source-data/dataset1` directory exists, and contains all the files of the resource group.

## Other kinds of groups

For downloading paginated APIs or partitioned datasets, see the dedicated pages:

- [Page resource groups](page-resource-groups.md) — enumerate pages with a stop criterion
- [Partition groups](partition-groups.md) — split large requests into smaller partitions