HTTP resources¶

The HttpResource class is a concrete implementation of the FileResource class.

This class uses requests to fetch URLs and tenacity to implement retrying.

Retry strategy¶

By default, the HttpResource is configured with sensible defaults regarding HTTP retries, implemented via the create_requests_retrying function.

Some HTTP response status codes trigger a retry, and a few connection exceptions also trigger a retry.

For example, a 404 (Not Found) does not trigger a retry, but a 429 (Too Many Requests) does, as well as a 503 (Service Unavailable).

The strategy applies a delay between retry attempts based on the Retry-After HTTP header in the response (cf RFC 6585), or if not found fallbacks by applying an exponential delay, starting from 1.5 seconds for the second attempt.

By default, a maximum of 5 attempts is made before giving up.

The retry strategy can be customized by passing the retrying kwarg to the constructor of the HttpResource class, or by creating a child class that overrides the HttpResource._default_retrying property.

You can also pass http_codes_to_retry to create_requests_retrying to customize which status codes trigger a retry.

Another way to customize the retry strategy is to raise the RetryHttpRequest exception (cf the following example).

Example: simulate a busy server with 200 status code¶

Sometimes the server responds something like “Server busy, retry later” but yet responds a status code of 200 OK.

In this case, the default retry strategy of HttpResource thinks that the response is successful and does not dig into its contents to determine whether or not to retry downloading the resource.

Let’s first simulate the server responses:

src/abc_fetcher/downloader.py¶

import responses

class AbcDownloader(BaseDownloader):
    # [...]

    @override
    def start(self) -> None:
        with responses.RequestsMock(assert_all_requests_are_fired=False) as rsps:
            # When calling `rsps.add` many times for the same URL, the `RequestsMock` will consume each one in order for each incoming request from the client.
            rsps.add(
                responses.GET,
                "https://abc-provider.com/data/catalog.json",
                body="<p>Server busy, retry later<p>",
                content_type="text/html",
                status=200,
            )
            rsps.add(
                responses.GET,
                "https://abc-provider.com/data/catalog.json",
                json=[
                    {"dataset_id": "BOP", "dataset_name": "Balance of payments"},
                    {"dataset_id": "GDP", "dataset_name": "Gross domestic product"},
                ],
                status=200,
            )
            super().start()

If we try to run the script at this point, we’ll have an InvalidMimeType exception because the HTML response does not match the extension of the catalog.json file name (cf MIME type validation), but no retry will be done.

Although we could customize the retry strategy, it’s better to customize the HTTP response validation directly to make it fail, by passing the validate_response kwarg to the constructor of HttpResource:

src/abc_fetcher/downloader.py¶

import responses
from dbnomics_toolbox.retry_utils.requests.errors import RetryHttpRequest
from requests import Response

class AbcDownloader(BaseDownloader):
    # [...]

    @override
    def _iter_resources(self) -> Iterator[BaseResource]:
        def validate_response(response: Response) -> None:
            response.raise_for_status()
            if "Server busy" in response.text:
                msg = "Server is busy"
                raise RetryHttpRequest(msg, response=response)

        yield HttpResource(
            id="catalog",
            request="https://abc-provider.com/data/catalog.json",
            target_file=self._source_data_dir_manager.catalog_file,
            validate_response=validate_response,
        )

In validate_response, raise_for_status does not raise an exception as the response code is 200. Raising RetryHttpRequest (inherited from requests.HTTPError) makes the resource download fail and let the request to be retried.

In contrast, just raising a requests.HTTPError would make the resource download fail, but would not let the request to be retried, as the response code is 200.

Validate the response¶

By default the HttpResource._validate_response method calls the Response.raise_for_status method, which raises an exception if the response status code is unsuccessful.

The response validation can be customized by passing the validate_response kwarg to the constructor of the HttpResource class, or by creating a child class that overrides the HttpResource._validate_response method.

Proxies¶

If you need to use a proxy, you pass the proxies argument to the HttpResource class:

from dbnomics_toolbox.fetcher_utils.resources.http_resource import HttpResource

proxies = {
    "http": "http://10.10.1.10:3128",
    "https": "http://10.10.1.10:1080",
}

HttpResource(
    proxies=proxies,
    request="http://example.org,
    target_file="test.txt",
)

Alternatively you can configure it once for an entire Session:

from dbnomics_toolbox.fetcher_utils.resources.http_resource import HttpResource
from requests import Session

proxies = {
    "http": "http://10.10.1.10:3128",
    "https": "http://10.10.1.10:1080",
}

with Session() as session:
    session.proxies.update(proxies)
    resource = HttpResource(
        request="http://example.org,
        session=session,
        target_file=target_file,
    )

Proxies can also be configured by using the standard environment variables http_proxy, https_proxy, no_proxy, and all_proxy, as documented by the Requests library.

User-Agent¶

By default, HttpResource sends a custom User-Agent header via the DEFAULT_USER_AGENT constant. You can override it by passing the user_agent kwarg to the constructor, or set it to None to send no custom User-Agent.

Response dump¶

The dump_response parameter controls when HTTP request/response pairs are saved to the debug directory for inspection:

"never" — never save dumps
"invalid" (default) — save dumps only for responses that fail validation
"valid" — save dumps only for responses that pass validation
"always" — always save dumps

Dumps are written as .http_dump.attempt_N.txt files alongside the resource’s debug file.

Session reuse¶

You can pass a requests.Session instance via the session parameter to reuse connections across multiple resources:

from requests import Session

with Session() as session:
    resource1 = HttpResource(
        request="https://example.org/data1.xml",
        session=session,
        target_file="data1.xml",
    )
    resource2 = HttpResource(
        request="https://example.org/data2.xml",
        session=session,
        target_file="data2.xml",
    )

Timeout¶

Pass the timeout parameter to set a timeout on the HTTP request. It accepts the same formats as the Requests timeout parameter:

HttpResource(
    request="https://example.org/data.xml",
    target_file="data.xml",
    timeout=30,  # 30 seconds for both connect and read
)

Streaming and encoding¶

By default stream=True, meaning the response body is read in chunks rather than loaded entirely in memory. The chunk_size parameter controls the chunk size passed to Response.iter_content.

When use_response_charset=True (default), the response body is re-encoded from the server’s declared encoding to UTF-8. You can also pass a string to force a specific source encoding. Set use_response_charset=False to write the raw response bytes as-is.

The target encoding defaults to UTF-8 and can be changed via the target_encoding parameter.

Advanced parameters¶

allow_redirects (default: True) — follow HTTP redirects
cert — client-side certificate (see Requests docs)
verify — SSL certificate verification (see Requests docs)
decoder_errors (default: "strict") — error handling scheme for the incremental decoder (see codecs)