I have a new server running rstudio-server and FIPS mode. The service starts, but any attempt to login returns this is the logs:
ERROR system error 74 (Bad message) [description: error:1C8000A5:Provider routines::illegal or unsupported padding mode, description: Failed sign-in - unable to decrypt password - error]; OCCURRED AT rstudio::core::Error rstudio::core::system::crypto::rsaPrivateDecrypt(const string&, std::string*) src/cpp/core/system/Crypto.cpp:542; LOGGED FROM: void rstudio::server::pam_auth::{anonymous}::doSignIn(const rstudio::core::http::Request&, rstudio::core::http::Response*) src/cpp/server/ServerPAMAuth.cpp:136
I do NOT get this error on any of the other 20 or so servers with almost identical installations, and the error goes away when I disable FIPS mode. Unfortunately, FIPS mode is required, and I cannot leave it disabled.
I have not found anything else online about this exact error. I did see something about forcing rstudio to use encryption level 2, which i think is rsa256 or something, but doing that did not help either.