# Partition groups

[](#BasePartitionGroup) extends [](#ResourceGroup) to download large datasets that must be split into multiple partitions (e.g., by time period, by region, etc.).

## Partitioning vs pagination

Partitioned download introduces the concept of **split**: **1 resource can give rise to N resources** if it fails.

- When a resource download fails with a recoverable error (e.g., [](#ResponseTooLarge) or [](#NoResultFound)), the partition can be split into smaller sub-partitions that are re-enqueued.
- The split state is persisted via [](#PartitionStateRepo), so resume mode can restore previously split partitions without re-downloading them.

If you just need to enumerate pages without splitting, use [Page resource groups](page-resource-groups.md) instead.

## How it works

1. The group is initialized with a list of `initial_partitions` (subclasses of `BasePartition`).
2. Partitions are placed in a queue and processed one by one.
3. For each partition, `_create_resource(partition, ...)` is called to create a [](#FileResource).
4. Partitions can be split dynamically (pre-split, resume-mode split, or runtime split).
5. If all partitions succeed, files are moved to the target directory (same atomic behavior as [](#ResourceGroup)).

## Constructor parameters

In addition to all parameters inherited from [](#ResourceGroup), [](#BasePartitionGroup) accepts:

- `initial_partitions` (`list[TPartition]`) — the initial list of partitions to download.

## Abstract method

- `_create_resource(partition, *, partition_depth, partition_num, resource_id) -> FileResource` — must be implemented by the subclass to create a [](#FileResource) for a given partition.

## Partition splitting strategies

### Pre-split

If `partition.should_pre_split` returns `True`, the partition is split via `partition.split()` **before** any download attempt. This is useful when you know in advance that a partition is too large.

### Resume-mode split

When the downloader runs in resume mode, [](#PartitionStateRepo) is checked for previously recorded splits.
If the partition was already split in a previous run, the sub-partitions are re-enqueued without downloading the parent partition again.

Tracking the state of failed partitions is necessary because no file can be saved when a partition fails, as for regular resources.

### Runtime split

A resource download can raise [](#EnqueueSubPartitions) with a list of sub-partitions. This is typically done in response to SDMX errors like [](#ResponseTooLarge) or [](#NoResultFound).

## Example

```python
from dbnomics_toolbox.fetcher_utils.partitions import PeriodRangePartition
from dbnomics_toolbox.fetcher_utils.resources import BasePartitionGroup, FileResource, ResourceId, SdmxResource
from dbnomics_toolbox.model.periods import Period


class MyPartitionGroup(BasePartitionGroup[PeriodRangePartition]):
    def _create_resource(
        self, partition: PeriodRangePartition, *, partition_depth: int, partition_num: int, resource_id: ResourceId
    ) -> "FileResource":
        return SdmxResource(
            id=resource_id,
            request=f"https://example.org/sdmx/data/?startPeriod={partition.min_value}&endPeriod={partition.max_value}",
            target_file=f"{partition.id}.xml",
        )


# Usage:
group = MyPartitionGroup(
    id="dataset1",
    initial_partitions=[
        PeriodRangePartition(Period.parse("2020"), Period.parse("2024"), max_length=3),
    ],
)
```