How to get all paths on ftp server

General
1

Hi,
I've tried to get all path on a ftp address but it fails.
I don't want to adapt my code to every change on a subfolder.
For example, some folders have this structure: Enterprise/yyyy but others are Enterprise/common/yyyy.
So, is there a package that can scan y retrieve all files from the main folder and subfolders?
I tried gemini but it can't make a code without using a lot of chunks according to every exception.
Thanks for your time and interest.

PS: I leave the horrible code from gemini. I dislike it.

library(RCurl)
library(stringr)


url_base <- "ftp://ftp.site.br/Enterprise/"
r_comp <- c()

rec_direct <- function(current_url) {

  routes_found <- c()
  
  list_elements_raw <- tryCatch({
    getURL(current_url, ftp.use.epsv = TRUE, dirlistonly = TRUE)
  }, error = function(e) {
    return("")
  })
 
  if (list_elements_raw == "") {
    return(routes_found)
  }
  
  elements <- str_split(list_elements_raw, "\n")[[1]]
  elements <- elements[elements != ""]
  
  for (element in elements) {
    if (str_detect(element, "/$")) {

      sub_url <- paste0(current_url, element)
        routes_found <- c(  routes_found rec_direct(sub_url))
    } else if (str_detect(element, "\\.zip$")) {

      routes_found <- c(routes_found paste0(current_url, element))
    }

  }
  
  return(routes_found)
}
2

While Gemini does make few syntax errors (at least 2 missing commas) and there are strange assumptions regarding returned values that probably make it fail, I don't think I can fault the recursive logic or error handling here.

You can find FTP file listing utility functions from few packages, e.g. rdwd::indexFTP(). Techincally you could just source a single file from github if you'd rather not install the package:

devtools::source_url("https://raw.githubusercontent.com/brry/rdwd/refs/heads/master/R/indexFTP.R")
#> ℹ SHA-1 hash of file is "189b5358cfa682cc53636f1c489858fbf3ccb8ea"

# stub to make indexFTP() happy
checkSuggestedPackage <- \(...) TRUE 
indexFTP(base = "ftp:/test.rebex.net/", quiet = TRUE, nosave = TRUE)
#>  [1] "/pub/example/imap-console-client.png"
#>  [2] "/pub/example/KeyGenerator.png"       
#>  [3] "/pub/example/KeyGeneratorSmall.png"  
#>  [4] "/pub/example/mail-editor.png"        
#>  [5] "/pub/example/mail-send-winforms.png" 
#>  [6] "/pub/example/mime-explorer.png"      
#>  [7] "/pub/example/pocketftp.png"          
#>  [8] "/pub/example/pocketftpSmall.png"     
#>  [9] "/pub/example/pop3-browser.png"       
#> [10] "/pub/example/pop3-console-client.png"
#> [11] "/pub/example/readme.txt"             
#> [12] "/pub/example/ResumableTransfer.png"  
#> [13] "/pub/example/winceclient.png"        
#> [14] "/pub/example/winceclientSmall.png"   
#> [15] "/pub/example/WinFormClient.png"      
#> [16] "/pub/example/WinFormClientSmall.png" 
#> [17] "/readme.txt"

Unless cross-platform portability is one of the priorities, I'd just use an external tool for this. For example lftp:

# lftp in WSL-hosted Ubuntu, called from Windows host
lftp_list <- \(site) sprintf('wsl lftp "%s" -e "find;quit;"', site) |> pipe() |> readLines()
lftp_list("ftp://test.rebex.net/")
#>  [1] "./"                                    "./pub/"                               
#>  [3] "./pub/example/"                        "./pub/example/KeyGenerator.png"       
#>  [5] "./pub/example/KeyGeneratorSmall.png"   "./pub/example/ResumableTransfer.png"  
#>  [7] "./pub/example/WinFormClient.png"       "./pub/example/WinFormClientSmall.png" 
#>  [9] "./pub/example/imap-console-client.png" "./pub/example/mail-editor.png"        
#> [11] "./pub/example/mail-send-winforms.png"  "./pub/example/mime-explorer.png"      
#> [13] "./pub/example/pocketftp.png"           "./pub/example/pocketftpSmall.png"     
#> [15] "./pub/example/pop3-browser.png"        "./pub/example/pop3-console-client.png"
#> [17] "./pub/example/readme.txt"              "./pub/example/winceclient.png"        
#> [19] "./pub/example/winceclientSmall.png"    "./readme.txt"

Or if you happen to have GDAL tools installed ( perhaps from OSGeo4W / QGIS installation), gdal vsi list /vsicurl/ftp://... can provide a list of files and directories as JSON, which jsonlite::fromJSON() is happy to simplify into a data frame:

# GDAL utils from OSGeo4W / QGIS installation
gdalvsi_list <- function(site) {
  sprintf('""C:/Program Files/QGIS 3.44.2/bin/gdal.exe" vsi list -l -R "/vsicurl/%s""', site) |> 
    pipe() |> 
    jsonlite::fromJSON()
}
gdalvsi_list("ftp://test.rebex.net/")
#>                                   name      type  size last_modification_date permissions
#> 1                                  pub directory     0   2023-03-31 00:00:00Z  d---------
#> 2                          pub/example directory     0   2023-03-31 00:00:00Z  d---------
#> 3  pub/example/imap-console-client.png      file 19156   2007-02-16 00:00:00Z  ----------
#> 4         pub/example/KeyGenerator.png      file 36672   2007-03-19 00:00:00Z  ----------
#> 5    pub/example/KeyGeneratorSmall.png      file 24029   2007-03-19 00:00:00Z  ----------
#> 6          pub/example/mail-editor.png      file 16471   2007-02-16 00:00:00Z  ----------
#> 7   pub/example/mail-send-winforms.png      file 35414   2007-02-16 00:00:00Z  ----------
#> 8        pub/example/mime-explorer.png      file 49011   2007-02-16 00:00:00Z  ----------
#> 9            pub/example/pocketftp.png      file 58024   2007-03-19 00:00:00Z  ----------
#> 10      pub/example/pocketftpSmall.png      file 20197   2007-03-19 00:00:00Z  ----------
#> 11        pub/example/pop3-browser.png      file 20472   2007-02-16 00:00:00Z  ----------
#> 12 pub/example/pop3-console-client.png      file 11205   2007-02-16 00:00:00Z  ----------
#> 13              pub/example/readme.txt      file   379   2023-09-19 00:00:00Z  ----------
#> 14   pub/example/ResumableTransfer.png      file 11546   2007-03-19 00:00:00Z  ----------
#> 15         pub/example/winceclient.png      file  2635   2007-03-19 00:00:00Z  ----------
#> 16    pub/example/winceclientSmall.png      file  6146   2007-03-19 00:00:00Z  ----------
#> 17       pub/example/WinFormClient.png      file 80000   2007-03-19 00:00:00Z  ----------
#> 18  pub/example/WinFormClientSmall.png      file 17911   2007-03-19 00:00:00Z  ----------
#> 19                          readme.txt      file   379   2023-09-19 00:00:00Z  ----------