Partition groups

BasePartitionGroup extends ResourceGroup to download large datasets that must be split into multiple partitions (e.g., by time period, by region, etc.).

Partitioning vs pagination

Partitioned download introduces the concept of split: 1 resource can give rise to N resources if it fails.

  • When a resource download fails with a recoverable error (e.g., ResponseTooLarge or NoResultFound), the partition can be split into smaller sub-partitions that are re-enqueued.

  • The split state is persisted via PartitionStateRepo, so resume mode can restore previously split partitions without re-downloading them.

If you just need to enumerate pages without splitting, use Page resource groups instead.

How it works

  1. The group is initialized with a list of initial_partitions (subclasses of BasePartition).

  2. Partitions are placed in a queue and processed one by one.

  3. For each partition, _create_resource(partition, ...) is called to create a FileResource.

  4. Partitions can be split dynamically (pre-split, resume-mode split, or runtime split).

  5. If all partitions succeed, files are moved to the target directory (same atomic behavior as ResourceGroup).

Constructor parameters

In addition to all parameters inherited from ResourceGroup, BasePartitionGroup accepts:

  • initial_partitions (list[TPartition]) — the initial list of partitions to download.

Abstract method

  • _create_resource(partition, *, partition_depth, partition_num, resource_id) -> FileResource — must be implemented by the subclass to create a FileResource for a given partition.

Partition splitting strategies

Pre-split

If partition.should_pre_split returns True, the partition is split via partition.split() before any download attempt. This is useful when you know in advance that a partition is too large.

Resume-mode split

When the downloader runs in resume mode, PartitionStateRepo is checked for previously recorded splits. If the partition was already split in a previous run, the sub-partitions are re-enqueued without downloading the parent partition again.

Tracking the state of failed partitions is necessary because no file can be saved when a partition fails, as for regular resources.

Runtime split

A resource download can raise EnqueueSubPartitions with a list of sub-partitions. This is typically done in response to SDMX errors like ResponseTooLarge or NoResultFound.

Example

from dbnomics_toolbox.fetcher_utils.partitions import PeriodRangePartition
from dbnomics_toolbox.fetcher_utils.resources import BasePartitionGroup, FileResource, ResourceId, SdmxResource
from dbnomics_toolbox.model.periods import Period


class MyPartitionGroup(BasePartitionGroup[PeriodRangePartition]):
    def _create_resource(
        self, partition: PeriodRangePartition, *, partition_depth: int, partition_num: int, resource_id: ResourceId
    ) -> "FileResource":
        return SdmxResource(
            id=resource_id,
            request=f"https://example.org/sdmx/data/?startPeriod={partition.min_value}&endPeriod={partition.max_value}",
            target_file=f"{partition.id}.xml",
        )


# Usage:
group = MyPartitionGroup(
    id="dataset1",
    initial_partitions=[
        PeriodRangePartition(Period.parse("2020"), Period.parse("2024"), max_length=3),
    ],
)