This is reminiscent of the problem posted in this thread. As @nirgrahamuk stated in his post, the tidyverse functions aren't ideal for such algorithm-based calculations.
You may have to write a C++ function using Rcpp as he did to gain some real performance benefits. I've tagged him in this post to see if he can offer any additional advice.