I am working on an app to identify rainfall trends of multiple time series using Modified Mann Kendall method. There are about 300 stations and each station has 50 yrs worth of data (daily rainfall data) which are converted to more variables like annual rainfall, monthly rainfall, rainy season rainfall etc. All those variables needs to be tested for trends but due to the complexity of the mathematical equation the runtime is over 4 hrs which is too long.
Equation which takes too much time is in the image below. (I have implemented it using 4 consecutive for loops)
Is there any method to simplify the equation or alternative functions to be used instead of for loops to speed up the runtime?
Reference paper for the modified mann-kendall method : https://www.sciencedirect.com/science/article/pii/S0022169407006865?ref=pdf_download&fr=RR-2&rr=9096de656873e0d2
Hi @DSWijesena, welcome to the community!
In addition to @martin.R's answer (which I don't have the background to evaluate), keep in mind that writing for loops is pretty much the slowest way to accomplish anything in R.
If you re-write your loops using lapply from base R or map from purrr, I would guess you'd easily shrink the 4-hr time to less than an hour for nearly no changes in your code.
The problem as you describe also falls into a class of problems that is easily parallelized. This can be done using several R packages, or you could do something like using sparklyr. Even though Spark is more effective the larger the cluster you run it on, even on a relatively modern laptop you'll get 4-8x parallelism due to having a multi-core processor.
Best,
Randy
1 Like
Addressing the shiny specific issue, the general way to deal with this for an app is to precompute everything and save it to disk, so that when the app is deployed, the interactive parts of the app are just reading from those files rather than recomputing everything on the fly. Without more information though, it's difficult to tell whether that would be suitable for your use case.
2 Likes
I actually considered the package but the method/ equations used weren't exactly the same.
Hey! If you’re specifically worried about the long-running calculation locking up the rest of the app then have you read about the new non-locking functionality introduced by ExtendedTask?
While that won’t help with runtimes of the order of multiple hours (or realistically multiple minutes) it’s not functionality I’ve seen talked about much in the shiny community.
