I did some digging and also emailed the people who made the raster package. It turns out I was reprojecting my raster by calling addRasterImage(). I hadn't realised the rasters I were returning were not the Web Map Mercator Projection.
However, fixing the projection issue didn't solve the memory usage as R would still reproject even when they were the required projections.
What I had to do was use the project=FALSE parameter to make sure R didn't waste its time.
addRasterImage(raster_image, colors = pal, project = FALSE)