The amazing sparklyr API provides methods form common dplyr generics (like mutate), some tbl_* and sdf_* functions. Sometimes its confusing to understand which one to use as they seem to do that same thing.
pivot_wider and sdf_pivot seem to provide similar functionality with former bring intuitive. Is the former a spark SQL functionality only?
tbl_cache and tbl_persist seem to do the same thing (both have scanty documentation).
Please guide a regular user about an understanding of appropriate situations to use the similar looking functions.
pivot_wider() is intended to provide the same interface tidyr provides and offers much more functionalities than sdf_pivot() does.
More generally, sdf_* family of functions are simply R wrappers for Spark DataSet API functions.
tbl_* family of functions are required S3 methods for implementing the dplyr backend for Spark dataframes. They should not really be considered as part of the user-facing API of sparkylr.
One key difference between *_cache and *_persist is the latter has a parameter for the level of persistence (none, memory, disk, etc).