httr2 request fails, ok in browser

[IanW]

I have a url that works fine in Google Chrome, but fails when used in httr2. The actual failure message varies from run to run, often just timing out ("less than 1 byte received in 600 seconds").

I've tried with and without the user agent string with no change.

Can someone explain what I'm doing wrong here please?

> url = "https://api.nasdaq.com/api/screener/stocks?tableonly=false&limit=25&exchange=AMEX&download=true"
> response <- 
+   httr2::request(url) %>%
+   httr2::req_user_agent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36") %>%
+   httr2::req_perform(verbosity = 3)
*  Host api.nasdaq.com:443 was resolved.
*  IPv6: (none)
*  IPv4: 23.64.34.95
*    Trying 23.64.34.95:443...
*  schannel: disabled automatic use of client certificate
*  Connected to api.nasdaq.com (23.64.34.95) port 443
*  using HTTP/1.x
-> GET /api/screener/stocks?tableonly=false&limit=25&exchange=AMEX&download=true HTTP/1.1
-> Host: api.nasdaq.com
-> User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36
-> Accept: */*
-> Accept-Encoding: deflate, gzip
-> 
*  Request completely sent off
*  schannel: server closed abruptly (missing close_notify)
*  closing connection #9
Error in `httr2::req_perform()`:
! Failed to perform HTTP request.
Caused by error in `curl::curl_fetch_memory()`:
! Failure when receiving data from the peer [api.nasdaq.com]
Run `rlang::last_trace()` to see where the error occurred.
> rlang::last_trace()
<error/httr2_failure>
Error in `httr2::req_perform()`:
! Failed to perform HTTP request.
Caused by error in `curl::curl_fetch_memory()`:
! Failure when receiving data from the peer [api.nasdaq.com]
---
Backtrace:
    ▆
 1. ├─... %>% httr2::req_perform(verbosity = 3)
 2. └─httr2::req_perform(., verbosity = 3)
 3.   └─base::tryCatch(...)
 4.     └─base (local) tryCatchList(expr, classes, parentenv, handlers)
 5.       └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
 6.         └─value[[3L]](cond)
>

[Gabor]

If I try this from the command line, I also get weird errors:

curl -v --head 'https://api.nasdaq.com/api/screener/stocks?tableonly=false&limit=25&exchange=AMEX&download=true'

[...]
* Request completely sent off
* HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)
* Connection #0 to host api.nasdaq.com left intact
curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)

With HTTP 1.1:

curl -v --http1.1 --head 'https://api.nasdaq.com/api/screener/stocks?tableonly=false&limit=25&exchange=AMEX&download=true'

* Request completely sent off
* Recv failure: Operation timed out
* LibreSSL SSL_read: LibreSSL/3.3.6: error:02FFF03C:system library:func(4095):Operation timed out, errno 60
* Closing connection
curl: (56) Recv failure: Operation timed out

[IanW]

Many thanks Gabor, you sent me down the path to solving this. I located a very similar problem report at Curl command freezes and does not work as expected · Issue #12532 · curl/curl · GitHub, and using that, fixed my problem using:

response <-
httr2::request(url) %>%
httr2::req_user_agent("Mozilla/5.0") %>%
httr2::req_headers("Accept-Encoding" = "identity") %>%
httr2::req_headers("Connection" = "Keep-Alive") %>%
httr2::req_perform(verbosity = 3)

[adamhsparks]

Many thanks to both @IanW and @Gabor for helping me solve this vexing issue. This thread popped up in my Posit discussion email summary just at the right time and helped me with this very same issue that was having with my package, {read.abares}, that I'd submitted to rOpenSci for review, https://github.com/ropensci/software-review/issues/667#issuecomment-2692552735.

Ian, your fix worked perfectly for me, thank you so much, both of you!