Hi Adrian! Welcome!
Have you tried inserting section separators when you have a regular R script file open? RStudio’s code options are contextual to some extent. In an R HTML file, as in an R Markdown file, Insert Section is replaced by Insert Chunk in the Code menu.
If you are editing the HTML portion of an R HTML document, then those separators won’t make sense (comments are bracketed with <!-- and -->, like in regular HTML). Meanwhile, document wide code regions already exist in the form of chunks (which can be folded and show up in the jump to menu like sections would). I suspect it would be rather complex to have a second tier of code regions within chunks.