Realtime data updates in Rstudio IDE

RStudio IDE
1

Is there any way to get realtime data updates inside Rstudio IDE, e.g. using mqtt and data frames!?

2

Not on the IDE as far as I know but you can certainly get mqtt messages on a shiny app for example, which could give you near real-time reactivity, also, if your data comes from a device that accepts serial connections you can take a look at this post

https://zhuhao.org/post/connect-arduino-chips-with-r/

3

Thank you so much for your answer.
Regarding the Shiny option I imagine that this could be an option for many.
For me, I am using mqtt in javascript already.
However, I would like to specifically perform on the fly data analysis using the R Studio IDE...as soon as the data is available to R Studio desktop.

4

I don't quite understand your specific use case, can you give a concrete example of your workflow or at least elaborate a little?

5

Note: Please bear in mind that I am not an expert in R or Shiny, and therefore I might not be aware of some of the existent packages or realtime data handling possibilities

Workflow:

  1. Data is being published in realtime using a MQTT broker
  2. this data is generated in a webapp/javascript using Tensorflow and other computations
  3. This data is also associated to some web visualizations
  4. when something in this web visualization is interesting or when something is interesting in the data it would be great to be notified and to do a manual analysis immediately, e.g. using R Studio IDE
  5. it would be great to have a MQTT client subscriber running in R Studio IDE (especially if it is not blocking the R Studio IDE UI). Additionally, it would be great to be able to update a dataframe in R Studio in realtime and have in this same nonblocking code a flag to start a R based analysis
  6. or to simply be able to use the functionalities of R Studio IDE an run my analysis on the data collected so far, e.g. on the dataframe being updated by the MQTT subscriber.
  7. I would of course prefer not to store data in a CSV, json or database, but instead to have a multithreaded R application running in R Studio IDE, non-ui-blocking, and with variables being updated in realtime

I hope I was clear enough in my explanation and that it makes sense at all...

6

If the analysis is going to be manual as you said I see no point on having the data loaded on the IDE on real time (And as far as I know this is not supported on the IDE), you could simply get notified and execute a script to load the data into your R session when you start doing the manual analysis.
Obviously I don't know enough about your actual application to make a fair judgment but it seems you are envisioning an unnecessarily complicated workflow.

7

Thank you so much for your feedback @andresrcs
Yes, it is true and I am currently already using a manual integration procedure anyway.
The reasoning for me is that the data analysis research and experimentation (and because I am using realtime messages and MQTT) should be a step that I could do at any time without the need to store data, other than transferring it directly from MQTT to R Studio IDE. Setting up Shiny can also be costly and an unnecessary step in this very specific case.

8

You could probably use the later package.

9

Unnecessary, probably, but costly, not necessarily, if you don't require a lot of computation power you can have it set up for free or for very little money.