I thought it might be useful to start a thread dedicated to shiny workflow and effective practices.
It would be interesting to hear other people's thoughts on things like file setup (app.R vs ui.R + server.R + global.R) and more general advice on how you organise/navigate through your code.
As apps grow larger and larger, I find it can be quite 'fiddly' to add/remove elements efficiently and a lot of time is spent locating chunks of code or searching for orphan commas and brackets that are causing errors.
It looks like using Shiny Modules would be a good solution for making larger apps more manageable and reproducible but I have yet to test it out.
I've made very very few apps (<15), but have used both app.R and ui.R+server.R+global.R.
For me, it is generally easier to just use app.R if the app I'm creating is smaller in size and easily navigated. For my larger projects, I like to separate the code out for easier navigation. For these types of projects, I typically include a run.R file as well that I can use to create a desktop icon or something similar so that I don't need to open R to run them.
Plan out the type of app. For a lot of applications, sometimes a htmlwidget in Rmd will do, and covers more cases as time goes on and javascript gets cooler. This is preferable as then you can publish to any server, not needing Shiny. But if it needs more server side interaction, I look at using flexdashboard to keep Rmd elements, but have shiny embedded. If its a big app though, its easier to keep track using a ui.R or a HTML template. I have a HTML template based on Gentella which I have tweaked for a company template, or if a client wants something embedded in their website make a new HTML template with their CSS, or finally if none of those are appropriate go to using an ui.R file.
I use a separate ui.R and server.R file, no global.R just source supporting scripts at the top or both ui.R and server.R - (source("functions.R")) No big reason why not to use global.R but I potentially change the functions.R into a small package later that supports modules.
I then work on the ui.R until I have a UI I like. Add any boilerplate such as tracking and shinyjs if any of those features are useful (love the visible/non-visible flags it has). It can be really hard to debug a ui.R if you get a bracket or comma in the wrong place, so I try to comment the end of big UI tags such as fluidRows() etc. As I'm adding outputX elements that are named I start adding the renderX elements to server.R to keep track of the reactive names.
ui.R Can be annoying when removing elements with trailing commas breaking stuff, so I try to put a br() or div() at the end of each so its easier to delete stuff. I'm also strict with code spacing, put one element per line and run it through CMD+I to auto-indent as often as possible. It can lead to very long lines :-/ Another tip is to use comments at top of major sections, and folding those comments sections in the RStudio interface for sections you are happy with.
Once the ui.R looks reasonable, I first make sure I have a working version of the server side logic outside of shiny, as its a lot easier to debug it there. Then I look to make some reactives() and eventReactives() as needed. I typically have a raw data reactive, then a processed data reactive that uses req() to depend on the raw data and the inputs, then maybe a final plot reactive if used by lots of output elements.
I try to ensure a req() or validate(need() is on every function, and always use NULL when data is invalid within functions.
Then start building the renderOutputs(), ideally keeping the code within simple as other logic is outside. Start running the app locally and checking how they look in the ui.R, probably tweak some columns() and widths.
I debug by making sure any reactive objects are assigned to normal R objects after the req() (raw_data <- raw_data()) which I find keeps things simple, and then use browser() and cat() and str() generously to check bits are doing what I expected them to do. The browser() stops the app right at the top of the function, and you can walk through errors and inspect state more easily.
If there is anything I repeat a lot or use again, will look to convert it into a Shiny module, and put it in a package or a script to source via functions.R. I generally try to move all the reactives() to the top of the function, and outputs to the bottom with logic in-between.
Its not over yet as sometimes when you deploy to the server there are issues, for instance if the versions of packages aren't available on shinyapps.io or your own server. For public apps shinyapps.io is great, for private I use a combo of containerit, Docker and googleComputeEngineR to deploy it to a VM (this workflow) For user authentication I use googleID, which is at the top of the ui.R and server.R before any data is fetched.
HTH - bit more involved than I initially meant to write!
Mark
All of the shiny apps I'm currently maintaining and developing are embedded in packages. The package directory follows the general format of
pkg_name/
- inst/
- Application/
- www/
- global.R
- server.R
- ui.R
- man
- R
Each package includes a launch_application function that uses the system.file utility to find the Application folder and start the app. (I'm sure there are better ways of doing this, but my organization as weird technical policies)
This isn't much different than sourcing a file with all of the utility functions. I like doing it because it encourages me to write documentation for my functions (which I often appreciate when I have to come back to do maintenance or debugging), and it lets me use the devtools package to perform checks and tests. Checks and tests are useful for preventing problems before the app ever gets to production.
I build a lot of UI elements through functions. I should probably use modules more than I do, but I haven't yet gotten comfortable with designing modules that handle unique and diverse data structures yet.
When actually coding the app, I break the app into functional sections based on the UI. For example, if I have multiple tabs, I will place all of the code in server.R relevant to that tab in one place in the code. My server.R file usually follows the structure
shinyServer(function(input, output, session){
# Global Variables ---------------------------------------
# Reactive Lists ------------------------------------------
# Reactive Values ---------------------------------------
# General Observers -----------------------------------
# Tab 1 -----------------------------------------------------
# Reactive Values ---------------------------------------
# Observers -----------------------------------------------
# Event Observers --------------------------------------
# Output Elements --------------------------------------
# Download Handlers ----------------------------------
# Repeat the above structure for each tab/grouping
# Output Options
})
A totally different way to approach large Shiny sites is to abandon menu tabs and use URL-based menus to load pages of the site one at a time. The advantages are a much lower memory footprint, faster loading, and much smaller namespaces. This is how php-based programs like WordPress work.
I've developed a skeleton for a multi-page, multi-user Shiny web site, with user authentication. You can see/download the code from https://github.com/open-meta/om_skeleton.
This breakdown is great, and I will definitely be stealing some of your ideas (especially commenting major ui sections and folding them when done with them)!
One thing I will say regarding this:
If you do use a global.R you can just source your "functions.R" file from there. I sourced my external files into both my ui.R and server.R files separately when I first started using shiny and switching to global.R has saved myself from a lot of errors where I changed something that is outside of the ui or server elements but forgot to change it in the other.
This is really helpful, thanks Mark! That html template is a beauty, I'll definitely be testing it out.
Great advice from everyone on workflow, I've already been putting a lot it into practice and am really seeing the benefit in writing modules and UI functions to handle repetitive code blocks.
I have been looking very closely to the radiant collection of UI packages. It looks very well organized with a standard UI all the way for all the packages. radiant creator is Vincent Nijs. https://github.com/radiant-rstats/radiant. All of these packages use the same underlying UI.
This is the folder structure for the radiant_basics package. Additionally, there are: radiant.data, radiant.design, radiant.model, and radiant.multivariate.
@nutterb Putting a shiny application inside a dedicated package is certainly one approach I am pursuing. Have you deployed these applications on an internal Shiny server or RStudio Connect server? At my organization I've always deployed apps using the typical ui.R / server.R or app.R constructs inside the Git repository for the app, which are then pushed to our internal shiny servers. I'm just curious how you would handle a deployment when the app is inside a package.
I'm not a great person to ask about deployment. I work on a project that only allows Windows based servers. This means Shiny Server and RStudio Connect servers are not an option for me, since they only operate on Linux distributions.
In the simple form, I use a .bat file to kick off an R script that initiates the applications, runs it on a designated port, and then people connect to the app using a URL that looks like http://servername:port.
This is a pretty poor deployment solution. It was just the first thing I could come up with that we could maintain with relative ease. If RStudio ever releases Shiny Server that can be run on a Windows machine, we might change the way we do things (hint, hint!)
AN YEAR LATER:
I have joined the discussion late but never too late to discuss best practices. Thanks to Paul for originating such a wonderful thread.
While a lot of good ideas are floated here I was wondering the best way to maintain relative file structure in a shiny server. If your shinyApp is built over many functions that are sourced from many directories and these functions use data lying in their current directories then what do you set as current directory for the shiny app when it is uploaded to the server.
In my case, I have to set the working directory to one up (parent of the shiny app director) and then all relative directories are codes in the app.R.
I found that wrapping Shiny modules in R6 objects is useful, especially if you have lots of data-dependent and user-dependent ui generated on-the-fly, want to store app state for each user, and benefit from OOP inheritance. The latter works if you have several similar, but not identical modules and want to avoid code duplication.
How it is structured:
There's a parent R6Class called ShinyModule that contains call, clone and serialize methods. This is constant and can be put in a package.
Every module is then defined as an R6Class, inheriting from ShinyModule. It usually contains an active property ui that is used to generate UI using private fields.
module's server part is usual, except that callModule is called via ShinyModule's call function. Module's R6 (self) is always passed to the server.
What it achieves:
Within module's server you can write to any fields of R6 object by reference without triggering reactivity. These fields can be accessed from outside the module.
Selected values from R6 can be serialized, saved to disk or DB, storing users app state and used to restore state of dynamically generated UI. Serialization is recursive, so sub-modules will be saved as well.
Data-dependent modules can be cloned with controls pre-calculated with their child modules, enhancing performance.
Our use case:
We've built an app that lets users create forecasts on arbitrary slices of data. There's a large dataset that can have models built for it's different subsets. Particularly, we are forecasting sales by region/channel. Users can add ui tabs for each slice, define slice's filters and model parameters, and then aggregate all their forecasts into one.
Their work (filters and model parameters) can be saved to disk or to the server and restored from a file.
Reprex will take some effort, will post if anybody is interested. EDIT: will post reprex on weekend
Oh this is what i want I think @ksavin . But have not been able to start working on R6 & ShinyModule. Without object-oriented programming experience do you think the R6 class can be grasped by a relatively new R programmer who is used to functional programming. Am new to shiny also but have managed to create an app that does a fair amount of user interaction including edits on DT objects. Could you refer to any resources that could be a good starting point for R6 / ShinyModule learning? And I wholly welcome and look forward to your code snippets and your documentation. That may go a long way to help us all.
Thought I would add something in here to expand to Connect. It seems like the deployment apps to that environment is how RStudio would like to push forward. I started a thread to discuss this and came up with some best practices. There are also some admin considerations for when an app requires an SLA vs. self administration, especially in a corporate environment. For version control, I commented on a separate thread on how we use Git.
