To build upon the previous answers, a R-hub blog post presented some strategies for "data outside your package" How to distribute data with your R package - R-hub blog
The example of rnaturalearth might be especially relevant for you as it's a geospatial one see https://twitter.com/southmapr/status/1262759210946682888
For #rnaturalearth I made 3 packages, 2 on CRAN, 1 not, rnaturalearth has methods and small example data, rnaturalearthdata has medium res data, rnaturalearthhires has hires data and is hosted by @rOpenSci because too big for CRAN.
Regarding permanent caching it is explained in Persistent config and data for R packages - R-hub blog you can use the rappdirs package, or if your package depends on R above version 4, you can use tools::R_user_dir().