Spark connection fails when multiple spark jobs are submitted

We have a shiny app that is deployed to Connect. Recently we found that when multiple users submit spark jobs at the same time, Spark can fail to connect for some of those jobs. We use Sparklyr::spark_connect() function to connect to Spark. Does anyone have a way to mitigate this?

We added a function to try to connect again for 15 times, each time delaying 20 seconds and changing the port number, but it did not help; the spark still fails to connect occasionally. We looked up the log but there's not much useful info as the app name is not created yet.


This topic was automatically closed 54 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.