Shiny download data with filter parameters

[kaijabean]

This is something of a brainstorming/best practices question. I have a scenario similar to the one described here, where the user can use a Shiny app to filter a dataset in various ways with different inputs, and then download the resulting filtered dataset.

I'm fine with the mechanics of getting that to happen, but what I would love to do is to save the current values of each filter as some sort of metadata: either as an additional file (maybe downloaded along with the dataset as a zip file?) or somehow embedded into the downloaded csv itself (at the bottom?). Currently, short of manually copying down the filter values, there's no way for the user to look back and determine which filter settings resulted in the dataset that they've downloaded.

Does anyone have ideas on how I might accomplish this, or have you seen something similar done? I guess another approach would be to use some sort of code system to store the filter values in the file name itself, but some of my filters have a lot (~100) of possible values, so that seems like it could get messy.

I'd appreciate any thoughts!

[nirgrahamuk]

I would go with the multifile zip option. You could represent the filter settings in a JSON file with a matching filename to the csv, and bundle it as a zip

[kaijabean]

Ok, yeah, I was leaning toward that. I'm not super familiar with JSON--can you explain why you'd do this over, say, a text file?

[nirgrahamuk]

JSON will give structured output out of the box, otherwise you would have to devise your own.

[kaijabean]

Okay, so I've tried to make a zip download, and it isn't working--whether or not the "include parameters" checkbox is clicked, and no matter what I specify for a filename, I get a download called "qwe_download.html" that seems to be an html version of the shiny app itself, not a csv or zip file of the data.

Here is a minimal (I think?) toy example. I think my problem is that I really don't understand what's going on with all the functions inside downloadHandler: I've adapted this code from examples I found online where people were using for-loops, but I don't think it's working for my case. (Note that here I've written to a .txt file instead of a .json, but I may change that later).

Any thoughts on where I might be tripping up?
Thank you!

library(DT)
library(zip)

ui <- fluidPage(
  fluidRow(
    column(4,
           downloadButton("download", "Download current data as csv", class = "btn-success"),
           checkboxInput("paramsInclude", "Include parameters?", value = FALSE)
    ),
    br(),
    DT::DTOutput("table1") # table output at the bottom
  )
)

server <- function(input, output, session) {
  tab <- data.frame(number = c(1:10), letter = letters[1:10])
  
  params <- data.frame(parameters = c("1 through 10", "a through j"))
  output$table1 <- DT::renderDT({tab})
  
  observeEvent(input$download, {
    if(input$paramsInclude){
      output$downloadData <- downloadHandler(
        filename = function() {
          paste("dataDownload.", gsub("-", "", Sys.Date()), ".zip")
        },
        content = function(fname) {
          fs <- c()
          tmpdir <- tempdir()
          setwd(tempdir())
          write.table(params, file = "params.txt", row.names = F)
          write.csv(tab, file = "data.csv", row.names = F)
          fs <- c("params.txt", "data.csv")
          zip(zipfile = fname, files = fs)
        },
        contentType = "application/zip"
      )
    }
  })
}


shinyApp(ui, server)

[nirgrahamuk]

server <- function(input, output, session) {
  tab <- data.frame(number = c(1:10), letter = letters[1:10])
  
  params <- data.frame(parameters = c("1 through 10", "a through j"))
  output$table1 <- DT::renderDT({tab})
  
      output$download <- downloadHandler(
        filename = function() {
          paste("dataDownload.", gsub("-", "", Sys.Date()), ".zip")
        },
        content = function(fname) {
         
          fs <- c()
          tmpdir <- tempdir()
          
          write.csv(tab, file = file.path(tmpdir,"data.csv"), row.names = F)
          fs <- "data.csv"
          if(input$paramsInclude){
            write.table(params, file = file.path(tmpdir,"params.txt"), row.names = F)
          fs <- file.path(tmpdir,c("params.txt", "data.csv"))
          }
          zipr(zipfile = fname, files = fs)
        },
        contentType = "application/zip"
      )
  }

[olibravo]

Why to use JSON? JSON is OK, but You can stay with R data structures - a list. I'd go a step further and define a class "filter" (S3 system would be enough). Then You can create an object of this class and store input settings used for filterring as well as any other information You need (timestamps, a user who created that filter etc.) Finally, store the filter in rds object (saveRDS() function) if you're not using a database.

The solution suggested above seems for me to be too complex when looking at your needs. A list and RDS would be 100% enough. No extra complexity with zipping, unzipping - just stay with native R structures.

[kaijabean]

I agree with you that native R structures would make a lot more sense here, and I would absolutely use them if I were doing this for myself. But I need to use JSON or text files or csv's or some other non-R structure because the app I'm making is going to be primarily used by people used to dealing with text files and spreadsheets: they don't necessarily code in R. So I need to make those files accessible.

Maybe I'll offer an option to download a list as an Rda or Rds object, though, because for R users you're right that that would be WAY more helpful.

[kaijabean]

Thanks so much! I'll try this out a bit later and let you know if it works.

[olibravo]

I got it. Make use of write.csv function or jsonlite package to achieve your goals.

[leungi]

I’ve done something similar with the use of Shiny Bookmark.

Concept:

1. create a reactive value list (eg, r_val) to store the various filter steps
2. use observeEvent() to detect filter input changes and store to r_val
3. use purrr::map() to create a corresponding list of filtered data based on the stored filters

# assume col is column to filter, associated with the input$filter is stored in r_val

map(r_val, ~filter(full_data, col >= .x))

To write out, add a downloadHandler() and use writexl::write_xlsx(), which lends itself nicely to list data, and will write out each item in the list into its own tab in .xlsx.

Hope that helps

[kaijabean]

@nirgrahamuk this works! Thank you!

I've been going through it trying to figure out what each line does, and I was really confused for a while with the tempdir() command and the two-step process of first writing each file and then zipping them.

What I thought should happen (and what I was going for in my original code):

• specify the names of R objects
• specify an output directory
• specify a file path for the file you're going to create, ending in .zip
• pass those R objects to the zip file command, which then generates a zip file with the specified file name and writes it out.

Now that I've looked this over, I think what is actually happening is this:

• zip() and zipr() don't work on R objects, they work on already-existing files.
• That explains why we write.csv and write.table on each of our objects into a temporary directory, and then tell zipr() to operate on those files we've just created
• We save the files into a temporary directory because we don't actually want to write out the individual csv/text files for the user, we just want to return one single zipped file for them.

I'm still confused about:

• Why the filename portion of the downloadHandler is a function instead of just a character string
• Why you put the if statement inside the output$download <- downloadHandler() command instead of outside (with two calls to downloadHandler()). I assume the former is more efficient? But it's not necessarily how I think of it intuitively.

Those both seem like things I know how to find the answers to, though, so no need to explain unless you particularly feel like it. Thanks so much for your help!

[kaijabean]

That's a really good idea, @leungi. I haven't used Shiny Bookmarks before, so I'll check them out.

[nirgrahamuk]

Hello, and yes, the most important aspect you understand which is that its existant files that we can zip, so thats great.
About the structure of the if() over changed behaviour for whether to include params or not. I think you always want a download handler for the download button regardless, so its easy to maintain code when you can see exactly what and where that function is. Then the if() just handles the small difference in cases, whether a second file is involved or not.

Filenames for output typically are best arrived at programatically so a filename function is optimal for that.
Ultimately the filename, if not adjusted by the user, will be what gets passed to content as fname, a simple character string.