For many questions to one source. Returns a list of envelopes the same length
as reqs and in the same order, so a caller zips the results back onto
whatever it asked about by position.
Usage
perform_many(
reqs,
source = "API",
max_active = 6,
progress = FALSE,
secret_query = NULL,
max_tries = 3
)Arguments
- reqs
A list of httr2 requests, each prepared with
req_defaults().- source
A friendly label for the service, used in the user-facing message. For example
"MyGene".- max_active
Maximum requests in flight at once.
- progress
Passed to
httr2::req_perform_parallel().FALSEby default, because the usual caller is a Shiny app that renders its own.- secret_query
A named list of query-string credentials, applied to every request in the batch at dispatch. See
redact_secrets().- max_tries
Total attempts per host group, including the first.
1disables the retry pass entirely.
Value
A list of envelopes, the same length and order as reqs. See
envelope().
Details
Each request must already have been through req_defaults(), the same as for
perform().
Same host, not a mix
httr2 applies req_throttle() and req_retry() across the whole list rather
than per request, so a throttled request to one host makes an unthrottled
request to another wait behind it. Requests are therefore grouped by host and
each group is dispatched separately, which keeps each host's throttle bucket
honest but means host groups run one after another.
Pass requests for one host. To query many different services at once, use process-level concurrency in the application instead.
Supply a throttle
httr2's own advice is never to perform in parallel without req_throttle(),
because it is otherwise very easy to flood a source with simultaneous
requests. req_defaults() takes a throttle argument and defaults its realm
to the request's host. Use it. Public biological data sources are typically
run on a research budget.
The breaker acts between batches, not inside one
A host with an open breaker is never dispatched to, and every one of its
requests comes back skipped. But once a batch is in flight, a failure on the
first request cannot short-circuit the rest, because they have already been
sent. Transport failures within a batch are recorded, so they take effect on
the next call rather than the current one.
Retry over failed indices, not the whole batch
httr2 documents req_retry() as not firing under req_perform_parallel(),
so this function drives its own: a pass after the first re-sends only the
requests that came back rate_limited, timeout, or a 5xx. A settled
answer (ok, no_data, any other 4xx) is never retried. Before a retry
pass, the host's recorded pause is honored (see ratelimit_wait()), set
from that pass's Retry-After when a 429 or 503 sent one, or an
exponential backoff otherwise. See transport_stats() to see what a call
actually dispatched.
Examples
breaker_reset()
reqs <- lapply(
c("BRCA1", "TP53"),
function(symbol) {
req_defaults(httr2::request(paste0("https://mock.test/gene/", symbol)))
}
)
httr2::with_mocked_responses(
function(req) {
httr2::response(
status_code = 200,
headers = list(`content-type` = "application/json"),
body = charToRaw('{"ok":true}')
)
},
vapply(perform_many(reqs, "MyGene"), function(res) res$status, character(1))
)
#> [1] "ok" "ok"