rm will remove all of the objects that are stored in your global environment (which may be what you want) but will not unload any of the packages that you have loaded. You can do both by restarting your R session in RStudio with the keyboard shortcut Ctrl+Shift+F10 which will totally clear your global environment of both objects and loaded packages.
EDIT: As @prosoitos correctly points out below, restarting your R session will only have the desired effects if you are not saving your workspace to your .Rdata file (which is typically not recommended)