The main reference for all things RMarkdown is https://bookdown.org/yihui/rmarkdown/
For output format specific information, consider https://bookdown.org/yihui/rmarkdown/pdf-document.html
In general, when you write RMarkdown, you then render that document to some format. In RStudio, that involved knitting the document (knit/knitr/compose notebook adn other names you may see for it). That takes the .Rmd, extracts the plain text to a .md file and the .R code to scripts. The R code is executed and the resulting output is merged into the .md file. That .md file is then handed to pandoc, a near-universal document converter that takes a given format as input and produces any one of hundreds of formats as output, including PDF (using a style template), HTML, MS Word (using OOXML), etc.
So, any particular style issue one might see could be a matter of how knitr processes the .Rmd, how R executes the .R code, how pandoc converts .md to .pdf or .docx or .html, or how the application you use to view the resulting document decides to render the file.