How to properly throttle a request using httr2?

General
,
1

Hi,

What am I doing wrong in this code so that the throttling is seemingly being ignored? Or what am I not understanding about how this should work? The rate limit for this API is 3 requests per second. But seemingly no matter what throttle capacity and file_time_s I use, it still errors from too many requests.

library(httr2)

foo <- function() {
  print('Trying request...')
  request("https://eutils.ncbi.nlm.nih.gov/entrez/eutils") |>
  req_url_path_append("esearch.fcgi") |>
  req_url_query(
    db = 'pubmed',
    term = 'Richard Feynman[au]',
    retmode = 'json'
  ) |>
  req_throttle(capacity = 1, fill_time_s = 60) |>
  req_perform() |>
  resp_body_json()
}

replicate(100, foo())
#> [1] "Trying request..."
#> [1] "Trying request..."
#> [1] "Trying request..."
#> [1] "Trying request..."
#> Error in `req_perform()`:
#> ! HTTP 429 Too Many Requests.
2

The issue is that your req_perform calls are not dealing with the same request object.

With each foo function call another unnamed request object is generated without waiting time.

Please see the examples provided in ?req_throttlefor its intended use.

Something like this should work:

library(httr2)

req <- request("https://eutils.ncbi.nlm.nih.gov/entrez/eutils") |>
  req_url_path_append("esearch.fcgi") |>
  req_url_query(
    db = 'pubmed',
    term = 'Richard Feynman[au]',
    retmode = 'json'
  ) |>
  req_throttle(capacity = 1, fill_time_s = 1)

foo <- function(req) {
  resp <- req_perform(req) |>
    resp_body_json()
  print(resp)
}

replicate(100, foo(req))
1 Like
3

Thanks, @ismirsehregal. That does indeed fix the issue, but this behaviour is a breaking change from how throttling worked in prior versions of {httr2}.

If anyone stumbles upon this thread, the issue is being tracked here: req_throttle does not throttle requests even when no tokens are available · Issue #801 · r-lib/httr2 · GitHub