I have a pile of urls where I want to know whether the url is still alive. Most of the status codes make sense compared to what I get when I put the url into a browser manually. I'm trying to automate the process with {httr2}. An example of where I'm getting conflicting results is https://BioRender.com/mjepmxz, which becomes https://app.biorender.com/citation/6a67bf7acd0190ede3d0e38a after being redirected.
If I use curl from the command line looking just at the headers, I see this:
$ curl -IL https://BioRender.com/mjepmxz
HTTP/2 301
date: Wed, 07 Oct 2026 19:39:05 GMT
content-type: text/html
content-length: 166
location: https://www.biorender.com/mjepmxz
set-cookie: _cfuvid=Z_1gdY3AO6kKWL7WwJsnfvsFaXdJBeAXY5vANnE_UAY-1791401945.0002923-1.0.1.1-DejmfK0B1eg8T9riD3JLrFnDrNgZSqDdX5jn6cHf..c; HttpOnly; SameSite=None; Secure; Path=/; Domain=biorender.com
cf-ray: a46f5eac3e347a92-DTW
cf-cache-status: BYPASS
cache-control: private
server: cloudflare
strict-transport-security: max-age=31536000; includeSubDomains; preload
content-security-policy: frame-ancestors 'self'
x-frame-options: SAMEORIGIN
x-wf-region: us-east-1
alt-svc: h3=":443"; ma=86400
HTTP/2 404
date: Wed, 07 Oct 2026 19:39:05 GMT
content-type: text/html; charset=utf-8
set-cookie: _cfuvid=CnKqmRiUBkf_RYA3CqgbHqLumKirp36FpxZwlxWXmoo-1791401945.30971-1.0.1.1-abttX60eNLZv8kvyvikPXt6a299V4WjRCHOLQpYGjKM; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.biorender.com
cf-ray: a46f5eae2a24b33c-DTW
cf-cache-status: HIT
cache-control: max-age=43200
etag: W/"ea296f1a6c1050ab7052c2d71770ff56"
last-modified: Tue, 06 Oct 2026 23:06:38 GMT
server: cloudflare
strict-transport-security: max-age=31536000; includeSubDomains; preload
vary: accept-encoding
content-security-policy: frame-ancestors 'self'
surrogate-key: www.biorender.com 621e6f1effebfe03881da9bd pageId:65bc9217ce634d2ec189e399 404req
x-frame-options: SAMEORIGIN
x-wf-region: us-east-1
alt-svc: h3=":443"; ma=86400
Using {httr2}, I am doing the following...
tryCatch(
{ request("https://BioRender.com/mjepmxz") %>%
req_options(
followlocation = TRUE,
ssl_verifypeer = 0,
ssl_verifyhost = 0
) %>%
req_method("HEAD") %>%
req_error(is_error = ~ FALSE) %>%
req_perform()
}, error = \(x){list(status_code = 404) }
) %>%
with(status_code)
Which gets me 404. If I use the url that the browser redirects to, I get 200:
tryCatch(
{ request("https://app.biorender.com/citation/6a67bf7acd0190ede3d0e38a") %>%
req_options(
followlocation = TRUE,
ssl_verifypeer = 0,
ssl_verifyhost = 0
) %>%
req_method("HEAD") %>%
req_error(is_error = ~ FALSE) %>%
req_perform()
}, error = \(x){list(status_code = 404) }
) %>%
with(status_code)
Any ideas what might be going on here or how to get the 200 result using {httr2} when I don't know the redirected address from the outset? I seem to be having similar problems with sites like https://pubmlst.org/fpsychrophilum.
When I use {chromote} I don't get any errors (but this approach seems to have a high false positive rate...
tryCatch( {
#open the "b"rowser
b <- ChromoteSession$new(wait_ = TRUE) # "b" stands for "browser"
Sys.sleep(runif(1, 2, 3))
b <- b$new_session(width = 1700, height = 1800, wait_ = TRUE)
b$Page$navigate("https://BioRender.com/mjepmxz", wait_ = TRUE)
},
error = function(cnd) {
e <- list() #intialize e before the try catch
e$errorText <- "caught_chrome_timeout_error"
return(e)
} )
I suspect all this has something to do with preventing people from doing what I'm trying to do, but am hoping there is a work around.
