This error appears when R does not have enough memory to continue the operation. Your data seems to take all your available memory. Via your OS, redo the operation and watch your memory usage.
You mention that "there are many images", perhaps you don't need to keep them all in memory to perform this operation?
Here's a doc on CRAN on Memory Limits in R.
I am not sure if you can give us a reproducible example (reprex) of your issue as a starting point, but an outline of what your code is doing might be useful to folks here giving advice on how to avoid taking up so much memory.