Perform many requests as one batch

For many questions to one source. Returns a list of envelopes the same length as reqs and in the same order, so a caller zips the results back onto whatever it asked about by position.

Usage

perform_many(
  reqs,
  source = "API",
  max_active = 6,
  progress = FALSE,
  secret_query = NULL
)

Arguments

reqs

A list of httr2 requests, each prepared with req_defaults().

source

A friendly label for the service, used in the user-facing message. For example "MyGene".

max_active

Maximum requests in flight at once.

progress

Passed to httr2::req_perform_parallel(). FALSE by default, because the usual caller is a Shiny app that renders its own.

secret_query

A named list of query-string credentials, applied to every request in the batch at dispatch. See redact_secrets().

Value

A list of envelopes, the same length and order as reqs. See envelope().

Details

Each request must already have been through req_defaults(), the same as for perform().

Same host, not a mix

httr2 applies req_throttle() and req_retry() across the whole list rather than per request, so a throttled request to one host makes an unthrottled request to another wait behind it. Requests are therefore grouped by host and each group is dispatched separately, which keeps each host's throttle bucket honest but means host groups run one after another.

Pass requests for one host. To query many different services at once, use process-level concurrency in the application instead.

Supply a throttle

httr2's own advice is never to perform in parallel without req_throttle(), because it is otherwise very easy to flood a source with simultaneous requests. req_defaults() takes a throttle argument and defaults its realm to the request's host. Use it. Public biological data sources are typically run on a research budget.

The breaker acts between batches, not inside one

A host with an open breaker is never dispatched to, and every one of its requests comes back skipped. But once a batch is in flight, a failure on the first request cannot short-circuit the rest, because they have already been sent. Transport failures within a batch are recorded, so they take effect on the next call rather than the current one.

Examples

breaker_reset()
reqs <- lapply(
  c("BRCA1", "TP53"),
  function(symbol) {
    req_defaults(httr2::request(paste0("https://mock.test/gene/", symbol)))
  }
)

httr2::with_mocked_responses(
  function(req) {
    httr2::response(
      status_code = 200,
      headers = list(`content-type` = "application/json"),
      body = charToRaw('{"ok":true}')
    )
  },
  vapply(perform_many(reqs, "MyGene"), function(res) res$status, character(1))
)
#> [1] "ok" "ok"