Hi, and welcome!
This problem doesn't require it, but keep in mind this FAQ: What's a reproducible example (`reprex`) and how do I create one? . Often answers without them go unanswered.
The best explanation is in the docs.
Error messages beginning cannot allocate vector of size indicate a failure to obtain memory, either because the size exceeded the address-space limit for a process or, more likely, because the system was unable to provide the memory. Note that on a 32-bit build there may well be enough free memory available, but not a large enough contiguous block of address space into which to map it.
I have been blessedly WIN-free for 10-years and can't offer specific tweaks. In the *nix world it is -ulimit.
If you are routinely going to be working with largish datasets that create this problem, you should consider renting EC2 instances with enough RAM to avoid this problem and do your pre-testing on a smaller sample.