I noticed that when I click the "Knit to PDF" or "Knit to Word" options in the Knit pulldown menu, RStudio will run rmarkdown::render_site() in any case, so I'm only getting HTML outputs. If I use rmarkdown::render() manually in the console, everythings work fine.
RStudio is probably using rmarkdown::render_site() instead of render() because you have named your file index.Rmd which is a naming convention for websites. Do you have a _site.yml file in the same directory?
Based on the pandoc command, it looks like it is still rendering out the word document --output index.docx, or at least named as if it were a Word doc.
It would use render_siteinstead of render if the file is called index.RmdAND there is a site: field with a render function. Otherwise, it would be if there is a _site.yml file present in the project root file as you said, and it that case the default rmarkdown website renderer would be used.