Hello,
I just want some advice. Currently I have a shiny app that has files uploaded (.csv) with it which read those files into the session upon launch. These are in chunks as they are relatively large. Would it be worthwhile for me to make use of the arrow package instead of "reading" them in with shiny or should arrow only be used when you in fact have a separate database somewhere else?
I wouldnt think arrow would be a significant help to you.
I'm guessing that the slowest part of getting the csv's in is because its not a data efficient format, maybe have your users zip their csv's for smaller file transfer times, and have you code unzip them in a local tempdir. Then use data.table fread to read the csv.
If the users are r-users and prepare the data as dataframes before exporting to csv, have them export them with library(fst) , and evolve your app to know how to read in fst files. my 2cents
Thank you for the reply 
I just thought of asking. At this point I am relying on vroom which seems to be decent. I was hoping that by potentially using something like arrow would "store" it slightly differently and maybe change how the memory of the instance is but I don't think it would be the case either. I just thought of asking at leas to check that I am not missing a good opportunity here.