Time Series Cross Validation Warning (fit_resamples)

I am trying to use tidymodels to run time series cross validation and have been following the Tidy Modeling with R chapters (10 Resampling for Evaluating Performance | Tidy Modeling with R).

When I run the following code, I get "Warning message: More than one set of outcomes were used when tuning. This should never happen. Review how the outcome is specified in your model."

The resamples$.metrics and resamples$.predictions are empty except for the first [[1]].

Any help is appreciated thank you!

library(tidyverse)
library(tidymodels)

df <- data.frame(
  year_month   =  seq(as.Date("2020-01-01"), as.Date("2024-03-01"), "month"),
  value  =  sample(1:100, length(seq(as.Date("2020-01-01"), as.Date("2024-03-01"), "month")))
)

splits <- sliding_window(df, lookback = Inf, skip=23, assess_start = 1, assess_stop = 1, complete = T)

model_spec <- arima_reg() %>%
  set_engine("auto_arima")

resamples <- fit_resamples(
  object       = model_spec,
  preprocessor = recipe(value ~ year_month, splits$splits[[1]]),
  resamples    = splits,
  control      = control_resamples(save_pred = TRUE)
)

collect_metrics(resamples)

resamples %>%
  select(id, .predictions) %>%
  unnest(.predictions)

Welcome to Posit Community, @nmontalbano!

I'm able to replicate the issue you're seeing with the CRAN version of tune. However, if I update my tune version to the current dev version (currently 1.1.2.9021) by running devtools::install_github("tidymodels/tune") and then restarting R, the warning about >1 outcomes goes away. That tune version will be on CRAN by the end of the month.

That said, some issues remain with this approach (note the number of rows in assessment(splits$splits[[1]]) and the message in show_notes(resamples)), but I'll let you troubleshoot from there. :slight_smile:

That worked, thank you for the welcome and the help!

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.