Colorised LaTEX formula

From your sreenshot, I suppose you aim at a HTML output ?

The behavior you encounter is because of the MathJax version used. By default, Rmarkdown is using version 2.7 and the syntax is not what you wrote.

See color — MathJax 3.2 documentation

in version 2, you would do
\color{red}{x} + \color{blue}{y}
to get a red x added to a blue y . But in version 3 (and in LaTeX itself), you would do
{\color{red} x} + {\color{blue} y}

You used the second.

This works:

---
title: "test"
output:
  html_document: 
    mathjax: default
---

$$
\color{Red}{\widetilde{\beta_{1}}}X_{1}
$$

You can also change the mathjax version I believe (but I don't know if there could be some unwanted behavior with some of the rmarkdown features.

This works

---
title: "test"
output:
  html_document: 
    mathjax: https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js
---

$$
{\color{Red} \widetilde{\beta_{1}}}X_{1}
$$

This does not

---
title: "test"
output:
  html_document: 
    mathjax: default
---

$$
{\color{Red} \widetilde{\beta_{1}}}X_{1}
$$

Hope it helps.