Fetching data from FTP server to shiny app in shinyapps.io

I'm trying to fetch some data from a FTP server to my shiny app, as follows:

library(RCurl)
library(shiny)
library(DT)


url <- getURL('ftp://ftpupload.net/myfolder/mydata.txt', userpwd='userlogin:password')
data <- read.delim(textConnection(url))

ui <- fluidPage(
  fluidRow(dataTableOutput('table'))
  )


server <- function(input, output) {
    output$table <- renderDataTable(
      datatable(data, options = list(paging = FALSE))
      )
}

shinyApp(ui = ui, server = server)

The app works perfectly in my local host, but when I upload it to shinyapps.io, the following error ocurrs when the app URL is acessed:

An error has occurred

Unable to connect to worker after 60.00 seconds; startup took too long.

Even when I set the Startup Timeout to 300 sec (in the app shinyapps.io advanced settings) i still having the connection time out error:

Error in value[3L] :
Failed to connect to ftpupload.net port 21: Connection timed out

The data to be fetched is a 206 Kb .txt file. Do someone know how to solve this problem? Any help will be appreciated.

I don't if it's related, but I'd generally recommend avoiding the RCurl package in favour of the newer curl package. It might at least give you better error messages.

Thank you for your reply Hadley. I updated my code as you suggest, using curl_fetch_memory from curl package, instead of getURL from rCurl, and now the script looks as follows:

library(curl)
library(shiny)
library(DT)

h <- new_handle(CONNECTTIMEOUT = 300)
handle_setopt(handle = h,  httpauth = 1,  userpwd = 'userlogin:password')
response <- curl::curl_fetch_memory("ftp://ftpupload.net/myfolder/mydata.txt", handle = h)
data <- rawToChar(resp$content) #Get raw data
data <- data.frame(x = strsplit(data , "\n", "", fixed = TRUE)[[1]]) #Separate in lines
data <- data %>% separate(x, "\t", into = c('Column A',	'Column B',	'Column C'), extra = "merge") #Separate in columns
data<- data[-1,] #Remove the first line

ui <- fluidPage(
  fluidRow(dataTableOutput('table'))
)


server <- function(input, output) {
  output$table <- renderDataTable(
    datatable(data, options = list(paging = FALSE))
  )
}

shinyApp(ui = ui, server = server)

It stills running perfectly in local host. Unfortunately the error stills ocurring in shinyapps.io. The following message is given:

An error has occurred

The application failed to start (exited with code 1).

`Registered S3 method overwritten by 'xts':
method from
as.zoo.xts zoo

Attaching package: ‘DT’

The following objects are masked from ‘package:shiny’:

dataTableOutput, renderDataTable

rgdal: version: 1.4-4, (SVN revision 833)
Geospatial Data Abstraction Library extensions to R successfully loaded
Loaded GDAL runtime: GDAL 2.2.2, released 2017/09/15
Path to GDAL shared files: /usr/share/gdal/2.2
GDAL binary built with GEOS: TRUE
Loaded PROJ.4 runtime: Rel. 4.9.2, 08 September 2015, [PJ_VERSION: 492]
Path to PROJ.4 shared files: (autodetected)
Linking to sp version: 1.3-1
Warning in fun(libname, pkgname) :
rgeos: versions of GEOS runtime 3.5.1-CAPI-1.9.1
and GEOS at installation 3.5.0-CAPI-1.9.0differ
rgeos version: 0.5-1, (SVN revision 614)
GEOS runtime version: 3.5.1-CAPI-1.9.1
Linking to sp version: 1.3-1
Polygon checking: TRUE

Error in value[3L] :
Failed to connect to ftpupload.net port 21: Connection timed out
Calls: local ... tryCatch -> tryCatchList -> tryCatchOne ->
Execution halted

Hi @sergio.costa,

shinyapps.io does not block ftp traffic, however in this instance you are attempting to use ftpupload.net which is flagged as a known bad site and access to it is rejected. Would it be possible for you to use a different FTP server for your connection?

Thanks!

1 Like

Hi @stevenolen, thanks for your reply!

Would you have any free ftp server that possibly wouldn't be blocked by shinyapps.io to recommend?

I don’t know what all your requirements are, but could you host your data somewhere like GitHub or Google Drive? It seems like free FTP sites are extremely likely to wind up blacklisted.

Also, both readr and base read.delim() can read tab-delimited text from a URL:

read.delim("https://gist.githubusercontent.com/jcblum/0b4be351e6b6eb7e300961d4635774ff/raw/f13c999b06b4c15e8ba42c6690216ab5b05baa1d/iana_example.tsv")

library(readr)
read_tsv("https://gist.githubusercontent.com/jcblum/0b4be351e6b6eb7e300961d4635774ff/raw/f13c999b06b4c15e8ba42c6690216ab5b05baa1d/iana_example.tsv")
4 Likes

Hi @sergio.costa,

I think @jcblum's comment is spot on. FTP is generally not the most effective way to serve files these days. If the file is static, you should be able to include the file itself in your app. If the file is not static, something like google drive, dropbox or amazon s3 if you need authentication could be potentials -- if you don't need authentication for fetching the file then I think the gist example provided would be superb!

Ok problem solved!
I hosted the data in github and now the app is working fine.

The data is being uploaded everyday by a third party and we need authentication, so github is not the best choice right now. Anyway, the problem was the FTP server.

I will investigate which is the best approach among those cited by @stevenolen.

Thank you both @jcblum and @stevenolen for your replyes!

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.