Prediction Intervals for Stacked model from stacks()

[darwinsbeard]

Is it possible to calculate prediction intervals from a tidymodels stacked model?

Working through the example from the stacks() package here yields the stacked frog model (which can be downloaded here for reprex) and the testing data:

data("tree_frogs")
tree_frogs <- tree_frogs %>%
  filter(!is.na(latency)) %>%
  select(-c(clutch, hatched))

set.seed(1)
tree_frogs_split <- initial_split(tree_frogs)
tree_frogs_train <- training(tree_frogs_split)
tree_frogs_test  <- testing(tree_frogs_split)

I tried to run something like this:
pi <- predict(tree_frogs_model_st, tree_frogs_test, type = "pred_int")

but this gives an error:
Error in UseMethod("stack_predict") : no applicable method for 'stack_predict' applied to an object of class "NULL"

Reading the documentation of stacks() I also tried passing "pred_int" in the opts list:
pi <- predict(tree_frogs_model_st, tree_frogs_test, opts = list(type = "pred_int"))

but this just gives: opts is only used with type = raw and was ignored.

For reference, I am trying to do a similar thing that is done in Ch.19 of Tidy Modeling with R book

lm_fit <- fit(lm_wflow, data = Chicago_train)
 predict(lm_fit, Chicago_test, type = "pred_int")

which seems to work fine for a single model fit like lm_fit, but apparently not for a stacked model?

Am I missing something? Is it not possible to calculate prediction intervals for stacked models for some reason?

[Max]

This is very difficult to do.

Even if glmnet produced a prediction interval, it would be a significant underestimate since it doesn’t know anything about the error in each of the ensemble members.

We would have to get the standard error of prediction from all of the models to compute it for the stacking model. A lot of these models don’t/can’t generate that standard error.

The alternative is the use bootstrapping to get the interval but you would have to bootstrap each model a large number of times to get the overall prediction interval.

Sorry :-/

[darwinsbeard]

Thank you Max! I figured it was something like that, and it does seem like a lot of effort so I'll just scrap the idea.