From your sreenshot, I suppose you aim at a HTML output ?
The behavior you encounter is because of the MathJax version used. By default, Rmarkdown is using version 2.7 and the syntax is not what you wrote.
See color — MathJax 3.2 documentation
in version 2, you would do
\color{red}{x} + \color{blue}{y}
to get a red x added to a blue y . But in version 3 (and in LaTeX itself), you would do
{\color{red} x} + {\color{blue} y}
You used the second.
This works:
---
title: "test"
output:
html_document:
mathjax: default
---
$$
\color{Red}{\widetilde{\beta_{1}}}X_{1}
$$
You can also change the mathjax version I believe (but I don't know if there could be some unwanted behavior with some of the rmarkdown features.
This works
---
title: "test"
output:
html_document:
mathjax: https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js
---
$$
{\color{Red} \widetilde{\beta_{1}}}X_{1}
$$
This does not
---
title: "test"
output:
html_document:
mathjax: default
---
$$
{\color{Red} \widetilde{\beta_{1}}}X_{1}
$$
Hope it helps.