The first icon is RStudio (no project) and the second one was the website project (the name went away when I closed it). The 3rd icon is the website project but open.
I'm working on a MacBook Pro (M4 2024 & macOS Sequoia 15.7) with RStudio version 2025.09.0+387 (and R version 4.5.1). I have the "minimize windows into application icon" setting for the dock set to "on".
I have tried looking at the com.apple.dock.plist file to no success (the dock plist looks identical for 2 pinned versions of RStudio).
I, too, have this issue (Intel Mac 2019, Sequoia 15.6) with RStudio version 2025.05.1+513 and R version 4.5.1. In my case, when I click on the icon that does not display a project name, the menus display "(empty)" and only the RStudio and Edit items contain anything to select. (None of these allow selection of a different project.) Do you see the same?
As far as I know there is currently no way to avoid this. Each instance of RStudio is a separate process. Ideally we'd have a single RStudio instance that supports multiple main windows as that would help with this as well as improve the performance of opening multiple instances of RStudio. Tracked with Second window startup is slower · Issue #12235 · rstudio/rstudio · GitHub.
I don't have the same issue if both windows (project and non-project) are open.
However, if I close the non-project window and leave the project window open, I can no longer open the non-project RStudio from the icon (even using ctrl+click -> Open).