# HTTP resources

The [](#HttpResource) class is a concrete implementation of the [](#FileResource) class.

This class uses [requests](https://requests.readthedocs.io/en/latest/) to fetch URLs and [tenacity](https://tenacity.readthedocs.io/en/latest/) to implement retrying.

## Retry strategy

By default, the [](#HttpResource) is configured with sensible defaults regarding HTTP retries, implemented via the [](#create_requests_retrying) function.

Some HTTP response status codes trigger a retry, and a few connection exceptions also trigger a retry.

For example, a 404 (Not Found) does **not** trigger a retry, but a 429 (Too Many Requests) does, as well as a 503 (Service Unavailable).

The strategy applies a delay between retry attempts based on the `Retry-After` HTTP header in the response (cf [RFC 6585](https://datatracker.ietf.org/doc/html/rfc6585#section-4)), or if not found fallbacks by applying an exponential delay, starting from 1.5 seconds for the second attempt.

By default, a maximum of 5 attempts is made before giving up.

The retry strategy can be customized by passing the `retrying` kwarg to the constructor of the [](#HttpResource) class, or by creating a child class that overrides the `HttpResource._default_retrying` property.

You can also pass `http_codes_to_retry` to [](#create_requests_retrying) to customize which status codes trigger a retry.

Another way to customize the retry strategy is to raise the `RetryHttpRequest` exception (cf the following example).

### Example: simulate a busy server with 200 status code

Sometimes the server responds something like "Server busy, retry later" but yet responds a status code of [200 OK](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/200).

In this case, the default retry strategy of `HttpResource` thinks that the response is successful and does not dig into its contents to determine whether or not to retry downloading the resource.

Let's first simulate the server responses:

```{code-block} python
:caption: src/abc_fetcher/downloader.py
import responses

class AbcDownloader(BaseDownloader):
    # [...]

    @override
    def start(self) -> None:
        with responses.RequestsMock(assert_all_requests_are_fired=False) as rsps:
            # When calling `rsps.add` many times for the same URL, the `RequestsMock` will consume each one in order for each incoming request from the client.
            rsps.add(
                responses.GET,
                "https://abc-provider.com/data/catalog.json",
                body="<p>Server busy, retry later<p>",
                content_type="text/html",
                status=200,
            )
            rsps.add(
                responses.GET,
                "https://abc-provider.com/data/catalog.json",
                json=[
                    {"dataset_id": "BOP", "dataset_name": "Balance of payments"},
                    {"dataset_id": "GDP", "dataset_name": "Gross domestic product"},
                ],
                status=200,
            )
            super().start()
```

If we try to run the script at this point, we'll have an `InvalidMimeType` exception because the HTML response does not match the extension of the `catalog.json` file name (cf [](./base-classes.md#mime-type-validation)), but no retry will be done.

Although we could customize the retry strategy, it's better to customize the HTTP response validation directly to make it fail, by passing the `validate_response` kwarg to the constructor of `HttpResource`:

```{code-block} python
:caption: src/abc_fetcher/downloader.py
import responses
from dbnomics_toolbox.retry_utils.requests.errors import RetryHttpRequest
from requests import Response

class AbcDownloader(BaseDownloader):
    # [...]

    @override
    def _iter_resources(self) -> Iterator[BaseResource]:
        def validate_response(response: Response) -> None:
            response.raise_for_status()
            if "Server busy" in response.text:
                msg = "Server is busy"
                raise RetryHttpRequest(msg, response=response)

        yield HttpResource(
            id="catalog",
            request="https://abc-provider.com/data/catalog.json",
            target_file=self._source_data_dir_manager.catalog_file,
            validate_response=validate_response,
        )
```

In `validate_response`, `raise_for_status` does not raise an exception as the response code is 200.
Raising `RetryHttpRequest` (inherited from [`requests.HTTPError`](https://requests.readthedocs.io/en/latest/api/#requests.HTTPError)) makes the resource download fail and let the request to be retried.

In contrast, just raising a [`requests.HTTPError`](https://requests.readthedocs.io/en/latest/api/#requests.HTTPError) would make the resource download fail, but would not let the request to be retried, as the response code is 200.

## Validate the response

By default the `HttpResource._validate_response` method calls the [`Response.raise_for_status`](https://requests.readthedocs.io/en/latest/api/#requests.Response.raise_for_status) method, which raises an exception if the response status code is unsuccessful.

The response validation can be customized by passing the `validate_response` kwarg to the constructor of the [](#HttpResource) class, or by creating a child class that overrides the `HttpResource._validate_response` method.

## Proxies

If you need to use a proxy, you pass the `proxies` argument to the [](#HttpResource) class:

```python
from dbnomics_toolbox.fetcher_utils.resources.http_resource import HttpResource

proxies = {
    "http": "http://10.10.1.10:3128",
    "https": "http://10.10.1.10:1080",
}

HttpResource(
    proxies=proxies,
    request="http://example.org,
    target_file="test.txt",
)
```

Alternatively you can configure it once for an entire `Session`:

```python
from dbnomics_toolbox.fetcher_utils.resources.http_resource import HttpResource
from requests import Session

proxies = {
    "http": "http://10.10.1.10:3128",
    "https": "http://10.10.1.10:1080",
}

with Session() as session:
    session.proxies.update(proxies)
    resource = HttpResource(
        request="http://example.org,
        session=session,
        target_file=target_file,
    )
```

Proxies can also be configured by using the standard environment variables `http_proxy`, `https_proxy`, `no_proxy`, and `all_proxy`, as documented by the Requests library.

See also: <https://requests.readthedocs.io/en/latest/user/advanced/#proxies>

## User-Agent

By default, [](#HttpResource) sends a custom `User-Agent` header via the [](#DEFAULT_USER_AGENT) constant. You can override it by passing the `user_agent` kwarg to the constructor, or set it to `None` to send no custom User-Agent.

## Response dump

The `dump_response` parameter controls when HTTP request/response pairs are saved to the debug directory for inspection:

- `"never"` — never save dumps
- `"invalid"` (default) — save dumps only for responses that fail validation
- `"valid"` — save dumps only for responses that pass validation
- `"always"` — always save dumps

Dumps are written as `.http_dump.attempt_N.txt` files alongside the resource's debug file.

## Session reuse

You can pass a [`requests.Session`](https://requests.readthedocs.io/en/latest/api/#requests.Session) instance via the `session` parameter to reuse connections across multiple resources:

```python
from requests import Session

with Session() as session:
    resource1 = HttpResource(
        request="https://example.org/data1.xml",
        session=session,
        target_file="data1.xml",
    )
    resource2 = HttpResource(
        request="https://example.org/data2.xml",
        session=session,
        target_file="data2.xml",
    )
```

## Timeout

Pass the `timeout` parameter to set a timeout on the HTTP request. It accepts the same formats as the [Requests `timeout` parameter](https://requests.readthedocs.io/en/latest/user/advanced/#timeouts):

```python
HttpResource(
    request="https://example.org/data.xml",
    target_file="data.xml",
    timeout=30,  # 30 seconds for both connect and read
)
```

## Streaming and encoding

By default `stream=True`, meaning the response body is read in chunks rather than loaded entirely in memory. The `chunk_size` parameter controls the chunk size passed to [`Response.iter_content`](https://requests.readthedocs.io/en/latest/api/#requests.Response.iter_content).

When `use_response_charset=True` (default), the response body is re-encoded from the server's declared encoding to UTF-8. You can also pass a string to force a specific source encoding. Set `use_response_charset=False` to write the raw response bytes as-is.

The target encoding defaults to UTF-8 and can be changed via the `target_encoding` parameter.

## Advanced parameters

- `allow_redirects` (default: `True`) — follow HTTP redirects
- `cert` — client-side certificate (see [Requests docs](https://requests.readthedocs.io/en/latest/user/advanced/#ssl-cert-verification))
- `verify` — SSL certificate verification (see [Requests docs](https://requests.readthedocs.io/en/latest/user/advanced/#ssl-cert-verification))
- `decoder_errors` (default: `"strict"`) — error handling scheme for the incremental decoder (see [`codecs`](https://docs.python.org/3/library/codecs.html#codecs.register_error))