# Page resource groups

[](#BasePageResourceGroup) extends [](#ResourceGroup) to download resources from paginated HTTP APIs where the total number of pages is unknown in advance.

## Pagination vs partitioning

Paginated download is a simple **enumeration**:

- Each page number produces exactly **one** resource.
- If a resource fails, it is **not** retried as smaller sub-pages.
- The only way to stop is the [](#PageOutOfBounds) stop criterion.

If you need to split a failed resource into smaller ones (e.g., when the server responds `ResponseTooLarge`), use [Partition groups](partition-groups.md) instead.

## How it works

1. Starting from `start_page_num` (default: 1), the group calls `_create_resource(page_num)` for each page number.
2. The returned resource is downloaded.
3. When a page is out of bounds, the resource must raise [](#PageOutOfBounds), which stops the iteration.
4. If all pages succeed, files are moved to the target directory (same atomic behavior as [](#ResourceGroup)).

## Constructor parameters

In addition to all parameters inherited from [](#ResourceGroup), [](#BasePageResourceGroup) accepts:

- `start_page_num` (`int | None`, default: `1`) — the page number to start from.

## Abstract method

- `_create_resource(page_num: int) -> FileResource` — must be implemented by the subclass to create a [](#FileResource) for a given page number.

## Detecting out-of-bounds pages

The subclass is responsible for detecting when a page is out of bounds. This is typically done by inspecting the HTTP response content in a `validate_response` callback and raising [](#PageOutOfBounds).

### Example

```python
from pathlib import Path
from requests import Response

from dbnomics_toolbox.fetcher_utils.resources import BasePageResourceGroup, HttpResource, PageOutOfBounds


class CatalogPageGroup(BasePageResourceGroup):
    def _create_resource(self, page_num: int) -> HttpResource:
        def validate_response(response: Response) -> None:
            response.raise_for_status()
            if response.json() == []:
                raise PageOutOfBounds(page_num)

        return HttpResource(
            id=f"catalog_page_{page_num}",
            request=f"https://example.org/api/catalog?page={page_num}",
            target_file=Path(f"catalog_page_{page_num}.json"),
            validate_response=validate_response,
        )


# Usage:
group = CatalogPageGroup(id="dataset1")
```