Rstudio Crashes whiles running imaging codes

It would be easier to help with a minimal reproducible example (reprex). See here for instructions.

Building a reprex using the "nuclei.tif" example file provided by EBImage, which is a stack with 4 frames, I get the same problem as you at the rmObjects() step. Since this image has 4 frames, this error is solved by having temp.rm be a list of length 4.

library(EBImage)

x <- readImage(system.file('images', 'nuclei.tif', package='EBImage'))
mask1 <- thresh(x, 10, 10, 0.05)

cols <- rainbow(3, alpha=0.3)[2] # Initialize the green color
pred <- ifelse(mask1, cols, "#00000000")

fixed.pred <- fillHull(mask1)
fixed.pred <- bwlabel(fixed.pred)
temp.rm <- replicate(4,names(which(table(fixed.pred)<200)), simplify = FALSE)
fixed.pred <- rmObjects(fixed.pred, temp.rm)

fixed.pred2 <- ifelse(fixed.pred, cols, "#00000000")

Created on 2022-04-17 by the reprex package (v2.0.1)

If your mask1 is also a stack, this is a likely explanation.