I've created a httr::GET request that is proving very hard to troubleshoot because the host just hangs forever so I've included a timeout to force it to stop after 10 seconds. I was getting the same behavior in python using requests until I added a 'user-agent' header, but changing the user-agent in httr::GET does not yield the same result.
Here is the R code that doesn't generate a response (times out after 10 seconds so that user isn't sitting around forever)
library(httr)
resp <- GET(
url = "https://www.cmegroup.com/",
user_agent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36'),
timeout(10),
verbose()
)
print(resp)
Here is the python code that yields a response immediately
import requests as r
resp = r.get(
"https://www.cmegroup.com/",
headers = {'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36'},
timeout = 10
)
resp.status_code
print(len(resp.content))