It seems I don't have the same font on my system
> systemfonts::match_font("Caveat")
$path
[1] "C:\\WINDOWS\\Fonts\\arial.ttf"
$index
[1] 0
$features
NULL
but I can reproduce the error you see in RStudio.
It is possible that RStudio does not use this ragg device when you run a code chunk
I can't help much with this, and I don't think this is a rmarkdown issue.
Does the ggplot code works when you run it in a R console ?