Can't access pins using pin_get(), but I see the pinned data in RStudio Connect

[Hlynur]

Hi folks,

I'm trying out the pins package in an attempt to keep things simple in a scheduled rmarkdown document on RStudio Connect.

My problem is that I can pin datasets, but I can't access what I've pinned with the pin_get() function.

I've attached a reprex, but I feel like it may not be too helpful. Despite the error being thrown when I pin mtcars as "mtcars_test", it actually gets pinned to my RStudio Connect "board".

library(pins)

#We pin it
pin(mtcars, name = "mtcars_test", board = "rsconnect")
#> Error in pin_download_one(p, ...): Client error: (404) Not Found. Failed to download remote file: https://rstudio-prod.northeurope.cloudapp.azure.com/connectrstudio-prod.northeurope.cloudapp.azure.com:443/connect/content/3/data.txt

#But this doesn't work
mtcars_test_1 <- pin_get(name = "mtcars_test", board = "rsconnect")
#> Error in pin_download_one(p, ...): Client error: (404) Not Found. Failed to download remote file: https://rstudio-prod.northeurope.cloudapp.azure.com/connectrstudio-prod.northeurope.cloudapp.azure.com:443/connect/content/3/data.txt
mtcars_test_1
#> Error in eval(expr, envir, enclos): object 'mtcars_test_1' not found

#And neither does this, using my RStudio Connect username
mtcars_test_2 <- pin_get(name = "hlynur.hallgrimsson/mtcars_test", board = "rsconnect")
#> Error in pin_download_one(p, ...): Client error: (404) Not Found. Failed to download remote file: https://rstudio-prod.northeurope.cloudapp.azure.com/connectrstudio-prod.northeurope.cloudapp.azure.com:443/connect/content/3/data.txt
mtcars_test_2
#> Error in eval(expr, envir, enclos): object 'mtcars_test_2' not found

^{Created on 2021-02-06 by the reprex package (v0.3.0)}

Creating the above reprex was the first and only time I've pinned "mtcars_test", so it's not remnants of another code being run. Despite the error in the reprex I can see the dataset in my connections tab, I just can't access it.

If I press the blue arrow to expand the dataset in the connections tab, I get this pop-up, but no further information.

Also, I can see it on RStudio Connect but I can't use pin_get() in an rmarkdown to access it.

Can anyone tell me what I'm doing wrong or what could be causing this behaviour?

Many thanks,
Hlynur

[Hlynur]

So, I tried running it using httr::with_verbose, as Hadley has suggested when running into these kinds of problems.

So I see the problem (which I guess I should've noticed in the error message, but didn't ) is the url.

-> GET /connectrstudio-prod.northeurope.cloudapp.azure.com:443/connect/content/2/data.txt HTTP/1.1
-> Host: rstudio-prod.northeurope.cloudapp.azure.com
-> User-Agent: libcurl/7.64.1 r-curl/4.3 httr/1.4.2

instead of the GET url being

/connect/content/2/data.txt

or

rstudio-prod.northeurope.cloudapp.azure.com:443/connect/content/2/data.txt

it's actually

/connectrstudio-prod.northeurope.cloudapp.azure.com:443/connect/content/2/data.txt

So, now I know what the problem is. But I have no idea how to fix it. Is this something that is the result of some mess up in our RStudio Connect setup?

Any help much appreciated.

[cole]

Great catch! I believe you are missing the board_register() step, perhaps?

rsconnect::servers()$name
pins::board_register_rsconnect(name = "rsconnect", server = "myservername", account = "cole")
pins::pin_get("username/mtcars_")

You can also look at which "board" you currently have registered with pins::board_list(). It looks to me like you are trying to pass the server URL to pin_get when it should be passed to board_register(), which is creating your problem

I hope that helps! Definitely let us know if you continue having trouble!

[Hlynur]

Thanks so much for the reply Cole.

It doesn't seem to be related to board_register(). Here's a reprex with the board_register part included.

library(pins)

rsconnect::servers()$name
#> [1] "rstudio-prod.northeurope.cloudapp.azure.com"
#> [2] "shinyapps.io"

board_register_rsconnect(name = "rsconnect", 
                         server = "rstudio-prod.northeurope.cloudapp.azure.com",
                         account = "hlynur.hallgrimsson")

#Note that pinning actually works, despite the error being shown
pin(mtcars, name = "mtcars", board = "rsconnect")
#> Error in pin_download_one(p, ...): Client error: (404) Not Found. Failed to download remote file: https://rstudio-prod.northeurope.cloudapp.azure.com/connectrstudio-prod.northeurope.cloudapp.azure.com:443/connect/content/5/data.txt

# I can see the pin.
pin_find("mtcars", board = "rsconnect")
#> # A tibble: 1 x 4
#>   name                       description board     type 
#>   <chr>                      <chr>       <chr>     <chr>
#> 1 hlynur.hallgrimsson/mtcars ""          rsconnect ""

# It's only in the GET request that the url gets mangled
pin_get(name = "hlynur.hallgrimsson/mtcars", board = "rsconnect")
#> Error in pin_download_one(p, ...): Client error: (404) Not Found. Failed to download remote file: https://rstudio-prod.northeurope.cloudapp.azure.com/connectrstudio-prod.northeurope.cloudapp.azure.com:443/connect/content/5/data.txt

^{Created on 2021-02-10 by the reprex package (v0.3.0)}

Here's the same code with the addition of outputs from running it in httr::with_verbose()

library(pins)

rsconnect::servers()$name
#> [1] "rstudio-prod.northeurope.cloudapp.azure.com"
#> [2] "shinyapps.io"

board_register_rsconnect(name = "rsconnect", 
                         server = "rstudio-prod.northeurope.cloudapp.azure.com",
                         account = "hlynur.hallgrimsson")

#Note that pinning actually works, despite the error being shown
pin(mtcars, name = "mtcars", board = "rsconnect")
#> Error in pin_download_one(p, ...): Client error: (404) Not Found. Failed to download remote file: https://rstudio-prod.northeurope.cloudapp.azure.com/connectrstudio-prod.northeurope.cloudapp.azure.com:443/connect/content/5/data.txt

# I can see the pin.
pin_find("mtcars", board = "rsconnect")
#> # A tibble: 1 x 4
#>   name                       description board     type 
#>   <chr>                      <chr>       <chr>     <chr>
#> 1 hlynur.hallgrimsson/mtcars ""          rsconnect ""

# It's only in the GET request that the url gets mangled
httr::with_verbose(pin_get(name = "hlynur.hallgrimsson/mtcars", board = "rsconnect"))
#> -> GET /connect/__api__/applications/?count=10000&filter=content_type:pin&search=mtcars HTTP/1.1
#> -> Host: rstudio-prod.northeurope.cloudapp.azure.com
#> -> User-Agent: libcurl/7.64.1 r-curl/4.3 httr/1.4.2
#> -> Accept-Encoding: deflate, gzip
#> -> Cookie: rscid-legacy=**I have removed the cookie info**
#> -> Accept: application/json, text/xml, application/xml, */*
#>   -> Date: Wed, 10 Feb 2021 02:24:32 GMT
#> -> X-Auth-Token: **I have removed this info**
#> -> X-Auth-Signature: **I have removed this info**
#>   -> X-Content-Checksum: **I have removed this info**
#>   -> Content-Type: application/json
#> -> 
#>   <- HTTP/1.1 200 OK
#> <- Server: nginx/1.19.6
#>   <- Date: Wed, 10 Feb 2021 02:24:32 GMT
#> <- Content-Type: application/json; charset=utf-8
#> <- Content-Length: 1023
#> <- Connection: keep-alive
#> <- Cache-Control: no-cache, no-store, must-revalidate
#> <- Expires: 0
#> <- Pragma: no-cache
#> <- X-Content-Type-Options: nosniff
#> <- X-Frame-Options: DENY
#> <- 
#>   -> GET /connectrstudio-prod.northeurope.cloudapp.azure.com:443/connect/content/5/data.txt HTTP/1.1
#> -> Host: rstudio-prod.northeurope.cloudapp.azure.com
#> -> User-Agent: libcurl/7.64.1 r-curl/4.3 httr/1.4.2
#> -> Accept-Encoding: deflate, gzip
#> -> Cookie: rscid-legacy=**I have removed this info**
#> -> Accept: application/json, text/xml, application/xml, */*
#>   -> Date: Wed, 10 Feb 2021 02:24:32 GMT
#> -> X-Auth-Token: **I have removed this info**
#> -> X-Auth-Signature: **I have removed this info**
#>   -> X-Content-Checksum: **I have removed this info**
#>   -> Content-Type: application/json
#> -> 
#>   <- HTTP/1.1 404 Not Found
#> <- Server: nginx/1.19.6
#>   <- Date: Wed, 10 Feb 2021 02:24:32 GMT
#> <- Content-Type: text/html
#>   <- Content-Length: 1781
#> <- Connection: keep-alive
#>   <- X-Content-Type-Options: nosniff
#> <- 
#>     Error in pin_download_one(p, ...) : 
#>     Client error: (404) Not Found. Failed to download remote file: https://rstudio-prod.northeurope.cloudapp.azure.com/connectrstudio-prod.northeurope.cloudapp.azure.com:443/connect/content/5/data.txt

As mentioned above the URL gets mangled. The actual url gets a "/connect" added in front of it.

I have opened a ticket with RStudio support, but as posting a reprex is much nicer here than in the email support, I'm just going to refer to this most recent post.

Thanks again for responding Cole.

[cole]

Interesting!! Thanks so much for the reprex! And I'm glad you opened a support ticket - this is definitely something we want to have addressed!! (And well done redacting secrets!)

In the meantime, do you mind also sharing the output of this? I'm wondering if one of your server URLs has been garbled, as that might explain some of the behavior too

rsconnect::servers()[,c("name", "url")]

[Hlynur]

Thanks again Cole.

Here is that output

rsconnect::servers()[,c("name", "url")]
#>                                          name
#> 1 rstudio-prod.northeurope.cloudapp.azure.com
#> 2                                shinyapps.io
#>                                                                   url
#> 1 https://rstudio-prod.northeurope.cloudapp.azure.com/connect/__api__
#> 2                                         https://api.shinyapps.io/v1

^{Created on 2021-02-10 by the reprex package (v0.3.0)}

[cole]

Awesome, thanks for the update @Hylynur!

I am suspicious that your /connect/ proxy is confusing the pins package, unfortunately I'm going to dig in a bit and see if we can isolate / reproduce / resolve this issue!

[cole]

Alright, so something is going awry here, but I was not able to reproduce the issue, so I don't think it is the "connect/" path that is the problem.

Do you mind sharing what version of pins you are using? Is it the latest? I'm going to try to see if I can find the support issue you opened!

[cole]

Ok, I found it!! If we don't get much further, we'll probably set up a call and see if we can make some progress that way

In any case, can you share the output of pins::pin_info("mtcars", board = "rsconnect") ?

[Hlynur]

Awesome. Thanks so much for your continued support on this Cole!

packageVersion("pins")
#> [1] '0.4.5'

pins::pin_info("mtcars", board = "rsconnect")
#> Warning in value[[3L]](cond): Error searching 'rsconnect' board:
#> Client error: (404) Not Found. Failed to download remote file:
#> https://rstudio-prod.northeurope.cloudapp.azure.com/connectrstudio-
#> prod.northeurope.cloudapp.azure.com:443/connect/content/5/data.txt
#> Error in if ("metadata" %in% colnames(entry) && nchar(entry$metadata) > : missing value where TRUE/FALSE needed

^{Created on 2021-02-11 by the reprex package (v0.3.0)}

Edit: I should add that I've been going back and forth between the CRAN version and the dev version. Here's that same error with the dev version

packageVersion("pins")
#> [1] '0.4.5.9000'

pins::pin_info("mtcars", board = "rsconnect")
#> Warning in value[[3L]](cond): Error searching 'rsconnect' board:
#> Client error: (404) Not Found. Failed to download remote file:
#> https://rstudio-prod.northeurope.cloudapp.azure.com/connectrstudio-
#> prod.northeurope.cloudapp.azure.com:443/connect/content/5/data.txt
#> Error in if ("metadata" %in% colnames(entry) && nchar(entry$metadata) > : missing value where TRUE/FALSE needed

^{Created on 2021-02-11 by the reprex package (v0.3.0)}

[cole]

Cool, that's the same version I'm using.

Ugh, sorry about that. Let's try this one. Do you have an API key for RStudio Connect? If so export it as an environment variable CONNECT_API_KEY, and:

remotes::install_github("rstudio/connectapi")
connectapi::connect(host = "https://rstudio-prod.northeurope.cloudapp.azure.com/connect/")
myguid <- get_content(client, limit = Inf) %>% filter(id == 5) %>% pull(guid) %>% .[[1]]

mypin <- content_item(client, myguid)
mypin$get_content_remote()

The goal here is to access the content directly with the API and make sure the metadata on the content is as we expect.

[Hlynur]

Here it is

Sys.setenv(CONNECT_API_KEY = "**********************************")
library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
library(connectapi)

client <- connect(host = "https://rstudio-prod.northeurope.cloudapp.azure.com/connect/")
#> Defining Connect with host: https://rstudio-prod.northeurope.cloudapp.azure.com/connect/
myguid <- get_content(client, limit = Inf) %>% filter(id == 5) %>% pull(guid) %>% .[[1]]
#> Warning: The `get_content` function is experimental and subject to change without warning in a future release
#> This warning is displayed once per session.

mypin <- content_item(client, myguid)
mypin$get_content_remote()
#> $guid
#> [1] "********-****-****-****-********a873"
#> 
#> $name
#> [1] "mtcars"
#> 
#> $title
#> NULL
#> 
#> $description
#> [1] "A table pin with 32 rows and 11 columns."
#> 
#> $access_type
#> [1] "acl"
#> 
#> $connection_timeout
#> NULL
#> 
#> $read_timeout
#> NULL
#> 
#> $init_timeout
#> NULL
#> 
#> $idle_timeout
#> NULL
#> 
#> $max_processes
#> NULL
#> 
#> $min_processes
#> NULL
#> 
#> $max_conns_per_process
#> NULL
#> 
#> $load_factor
#> NULL
#> 
#> $created_time
#> [1] "2021-02-09T15:12:26Z"
#> 
#> $last_deployed_time
#> [1] "2021-02-10T02:24:31Z"
#> 
#> $bundle_id
#> [1] "35"
#> 
#> $app_mode
#> [1] "static"
#> 
#> $content_category
#> [1] "pin"
#> 
#> $has_parameters
#> [1] FALSE
#> 
#> $r_version
#> NULL
#> 
#> $py_version
#> NULL
#> 
#> $run_as
#> NULL
#> 
#> $run_as_current_user
#> [1] FALSE
#> 
#> $owner_guid
#> [1] "********-****-****-****-********dbdc"
#> 
#> $url
#> [1] "https://rstudio-prod.northeurope.cloudapp.azure.com:443/connect/content/5/"
#> 
#> $app_role
#> [1] "owner"

^{Created on 2021-02-11 by the reprex package (v0.3.0)}

[cole]

Flawless, as I feared My next thesis is something to do with port 443 being specified... but I will need to toy with that a bit to see if it matters. Hmm...

[cole]

We got to the bottom of this issue, and have a proposed fix here:

Fix urls

I was hoping we would have it merged quickly, but it may take a bit to get it onto master, so anyone who runs into this - feel free to use the proposed fix on my branch at present!