It's difficult to give a general answer, as that really depends on what you're doing. Even if the final raster image is small, maybe some intermediate steps are using up a lot of memory.
When running this code on your computer, can you see how much memory it uses? You can try using {profvis}, which gives you some estimates of memory, or just use your OS's task manager/resource monitor to look at what's happening. Both of these methods are not fully accurate, but give you an idea.
When I run your reprex code on my computer with profvis (not in a Shiny app), it looks like it's using 4.5 GB (but freeing up 3.5 GB), of these 3 GB in the call to addRasterImage(). So, following this, I tried adding project = FALSE, and indeed it strongly reduced the amount of memory needed (to 15 MB). This is also discussed here. Maybe that would help in your case?