Recurring work with an API

I'm trying to download a series of data from an api. This only allows downloading 100 items in each query, so I would like to do the query recursively until I have downloaded all the data. I download the data with the httr get command, but with that limitation I have to repeat it several times and segmented by date.

Any help would be appreciated

Conceptually, since the specifics are unknown, use a loop with a pause

testit <- function(x)
  p1 <- proc.time()
  proc.time() - p1 # The cpu usage should be negligible
#>    user  system elapsed 
#>   0.000   0.000   3.705

Created on 2023-11-09 with reprex v2.0.2

Sorry for the lack of information. The API is from:

raw_data <- GET("")

where display_name is the name of a user

You might find CRAN - Package OpenStreetMap helpful.

If it won't work for this particular request, you might also want to check out the dev version of {httr2} (an update is coming soon to CRAN), specifically req_perform_iterative().

I hope that helps!


# only have the one display_name to work with
users <- c("siebh","siebh")
raw_data <- vector(length = length(users))
for(i in seq_along(users)){
  raw_data[i] = list(
  Sys.sleep(0.5) # argument depends on server timeout policy

#> [[1]]
#> Response []
#>   Date: 2023-11-11 07:50
#>   Status: 200
#>   Content-Type: application/json; charset=utf-8
#>   Size: 36.2 kB
#> [[2]]
#> Response []
#>   Date: 2023-11-11 07:50
#>   Status: 200
#>   Content-Type: application/json; charset=utf-8
#>   Size: 36.2 kB

Created on 2023-11-10 with [reprex v2.0.2]

@jonthegeek thanks for suggestions. Package OpenStreetMap is for display raster maps, don't see any option to this work.

I'll give a look to httr2

@technocrat this seems to work but, when I try to make a df with the list of changesets, I get this error:

list <- fromJSON(rawToChar(raw_data), flatten = TRUE)
Error in rawToChar(raw_data) : argument 'x' must be a raw vector

The elements of raw_data are heterogenous, and only one of them is a suitable argument to rawToChar(). This snippet will show where to look.

# select first element from raw_data

f <- raw_data[1]

f[[1]]$url |> class()
f[[1]]$cookies |> class()
f[[1]]$status_code |> class()
f[[1]]$headers |> class()
f[[1]]$all_headers |> class()
f[[1]]$content |> class()
f[[1]]$date |> class()
f[[1]]$times |> class()
f[[1]]$request |> class()
f[[1]]$handle |> class()

# get the raw class
r <- f[[1]]$content

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.