Page resource groups

BasePageResourceGroup extends ResourceGroup to download resources from paginated HTTP APIs where the total number of pages is unknown in advance.

Pagination vs partitioning

Paginated download is a simple enumeration:

  • Each page number produces exactly one resource.

  • If a resource fails, it is not retried as smaller sub-pages.

  • The only way to stop is the PageOutOfBounds stop criterion.

If you need to split a failed resource into smaller ones (e.g., when the server responds ResponseTooLarge), use Partition groups instead.

How it works

  1. Starting from start_page_num (default: 1), the group calls _create_resource(page_num) for each page number.

  2. The returned resource is downloaded.

  3. When a page is out of bounds, the resource must raise PageOutOfBounds, which stops the iteration.

  4. If all pages succeed, files are moved to the target directory (same atomic behavior as ResourceGroup).

Constructor parameters

In addition to all parameters inherited from ResourceGroup, BasePageResourceGroup accepts:

  • start_page_num (int | None, default: 1) — the page number to start from.

Abstract method

  • _create_resource(page_num: int) -> FileResource — must be implemented by the subclass to create a FileResource for a given page number.

Detecting out-of-bounds pages

The subclass is responsible for detecting when a page is out of bounds. This is typically done by inspecting the HTTP response content in a validate_response callback and raising PageOutOfBounds.

Example

from pathlib import Path
from requests import Response

from dbnomics_toolbox.fetcher_utils.resources import BasePageResourceGroup, HttpResource, PageOutOfBounds


class CatalogPageGroup(BasePageResourceGroup):
    def _create_resource(self, page_num: int) -> HttpResource:
        def validate_response(response: Response) -> None:
            response.raise_for_status()
            if response.json() == []:
                raise PageOutOfBounds(page_num)

        return HttpResource(
            id=f"catalog_page_{page_num}",
            request=f"https://example.org/api/catalog?page={page_num}",
            target_file=Path(f"catalog_page_{page_num}.json"),
            validate_response=validate_response,
        )


# Usage:
group = CatalogPageGroup(id="dataset1")