Recurring work with an API

[jynusmac]

I'm trying to download a series of data from an api. This only allows downloading 100 items in each query, so I would like to do the query recursively until I have downloaded all the data. I download the data with the httr get command, but with that limitation I have to repeat it several times and segmented by date.

Any help would be appreciated

[technocrat]

Conceptually, since the specifics are unknown, use a loop with a pause

testit <- function(x)
{
  p1 <- proc.time()
  Sys.sleep(x)
  proc.time() - p1 # The cpu usage should be negligible
}
testit(3.7)
#>    user  system elapsed 
#>   0.000   0.000   3.705

^{Created on 2023-11-09 with reprex v2.0.2}

[jynusmac]

Sorry for the lack of information. The API is from:

raw_data <- GET("http://api.openstreetmap.org/api/0.6/changesets?display_name=xxxxxx")

where display_name is the name of a user

[jonthegeek]

You might find CRAN - Package OpenStreetMap helpful.

If it won't work for this particular request, you might also want to check out the dev version of {httr2} (an update is coming soon to CRAN), specifically req_perform_iterative().

I hope that helps!

[technocrat]

library(httr)

# only have the one display_name to work with
users <- c("siebh","siebh")
raw_data <- vector(length = length(users))
for(i in seq_along(users)){
  raw_data[i] = list(
    GET(
      paste0(
        "http://api.openstreetmap.org/api/0.6/changesets?display_name=",users[i])
      )
  )
  Sys.sleep(0.5) # argument depends on server timeout policy
}

raw_data
#> [[1]]
#> Response [https://www.openstreetmap.org/api/0.6/changesets?display_name=siebh]
#>   Date: 2023-11-11 07:50
#>   Status: 200
#>   Content-Type: application/json; charset=utf-8
#>   Size: 36.2 kB
#> 
#> 
#> [[2]]
#> Response [https://www.openstreetmap.org/api/0.6/changesets?display_name=siebh]
#>   Date: 2023-11-11 07:50
#>   Status: 200
#>   Content-Type: application/json; charset=utf-8
#>   Size: 36.2 kB

^{Created on 2023-11-10 with [reprex v2.0.2]}

[jynusmac]

@jonthegeek thanks for suggestions. Package OpenStreetMap is for display raster maps, don't see any option to this work.

I'll give a look to httr2

[jynusmac]

@technocrat this seems to work but, when I try to make a df with the list of changesets, I get this error:

list <- fromJSON(rawToChar(raw_data), flatten = TRUE)
Error in rawToChar(raw_data) : argument 'x' must be a raw vector

[technocrat]

The elements of raw_data are heterogenous, and only one of them is a suitable argument to rawToChar(). This snippet will show where to look.

# select first element from raw_data

f <- raw_data[1]

f[[1]]$url |> class()
f[[1]]$cookies |> class()
f[[1]]$status_code |> class()
f[[1]]$headers |> class()
f[[1]]$all_headers |> class()
f[[1]]$content |> class()
f[[1]]$date |> class()
f[[1]]$times |> class()
f[[1]]$request |> class()
f[[1]]$handle |> class()

# get the raw class
r <- f[[1]]$content
library(jsonlite)
fromJSON(rawToChar(r))