It is poorly documented how to set the java version being used in RStudio on the Mac. This leads to all sorts of problems, notably missing dynamically loaded libraries when loading the rJava package.
This is controlled by the system environment variables being used which can be checked with Sys.getenv() in the R terminal within RStudio.
On the Mac these variables will depend on how RStudio is launched. See macos - Setting environment variables on OS X - Stack Overflow for details. An example of what worked on my system to change these when launched from Spotlight were two launchctl commands:
launchctl setenv JAVA_HOME /Library/Java/JavaVirtualMachines/jdk-21.jdk/Contents/Home
sudo launchctl config user path JAVA_HOME /Library/Java/JavaVirtualMachines/jdk-21.jdk/Contents/Home
This issue is also addressed for the PATH variable at Where does the PATH variable come from in sys.getenv()?