The default Rmarkdown engine
/usr/local/bin/pandoc +RTS -K512m -RTS radix.utf8.md --to latex --from markdown+autolink_bare_uris+ascii_identifiers+tex_math_single_backslash --output radix.tex --template /Library/Frameworks/R.framework/Versions/3.6/Resources/library/rmarkdown/rmd/latex/default-1.17.0.2.tex --highlight-style tango --pdf-engine pdflatex --variable graphics=yes --variable 'geometry:margin=1in' --variable 'compact-title:yes'
is pdflatex.
30\textdegree
will work in a tex file using the XeLaTeX engine (provided your system font defaults contain the degree character). So, the question is how are you getting from the Rmd file to the tex file and what do you have in the preface to your tex file. Please post a reproducible example, called a reprex