RequestLoader

An abstract class defining the interface for classes that provide access to a read-only stream of requests.

Request loaders are used to manage and provide access to a storage of crawling requests.

Key responsibilities:

• Fetching the next request to be processed.
• Marking requests as successfully handled after processing.
• Managing state information such as the total and handled request counts.

Hierarchy

• RequestLoader
  • RequestList
  • RequestManager
  • SitemapRequestLoader

Index

Methods

• fetch_next_request
• get_handled_count
• get_total_count
• is_empty
• is_finished
• mark_request_as_handled

Methods

fetch_next_request

• async fetch_next_request(): Request | None

• Overrides RequestManager.fetch_next_request

  Return the next request to be processed, or null if there are no more pending requests.

  ---

  Returns Request | None

get_handled_count

• async get_handled_count(): int

• Overrides RequestManager.get_handled_count

  Get the number of requests in the loader that have been handled.

  ---

  Returns int

get_total_count

• async get_total_count(): int

• Overrides RequestManager.get_total_count

  Get an offline approximation of the total number of requests in the loader (i.e. pending + handled).

  ---

  Returns int

is_empty

• async is_empty(): bool

• Overrides RequestManager.is_empty

  Return True if there are no more requests in the loader (there might still be unfinished requests).

  ---

  Returns bool

is_finished

• async is_finished(): bool

• Overrides RequestManager.is_finished

  Return True if all requests have been handled.

  ---

  Returns bool

mark_request_as_handled

• async mark_request_as_handled(request): ProcessedRequest | None

• Overrides RequestManager.mark_request_as_handled

  Mark a request as handled after a successful processing (or after giving up retrying).

  ---

  Parameters

  • request: Request

  Returns ProcessedRequest | None

: