Is it possible to tune arbitrary parameters w/tidymodels?

[markjrieke]

tune is great & super useful for tuning model parameters - is it possible to also tune other parameters that aren't specified by a model (e.g., tune features rather than parameters)? The tune() documentation makes it seem like it may be possible, but I'm not 100% sure.

An example of this may be for predicting the winner of a game in sports & using each team's ELO rating as a predictor. It may be useful to tune the k-factor, rather than use the sport's ~generally recognized~ factor. I'm imagining something like this:

sports_rec <- 
  recipe(winner ~ ., data = sports_train) %>%
  step_mutate(ELO = rating + tune("k_factor") * (score - expected))

This example would definitely fail because I think it'd need to be a summary statistic, but the main takeaway would be trying to tune something outside of a model's predefined parameters.

I don't have a specific use case for anything I'm working on, but can imagine how that may be useful in the future. I figured this question was too general for stack overflow, but still may be interesting!

[markjrieke]

I suppose another way to do this would be to manually create a list where one col is the range for the tuned parameters and another col contains the training df(s), like so:

library(tibble)

tibble(param = seq(0, 1, 0.1),
       data = list(as_tibble(mtcars)))
#> # A tibble: 11 x 2
#>    param data              
#>    <dbl> <list>            
#>  1   0   <tibble [32 x 11]>
#>  2   0.1 <tibble [32 x 11]>
#>  3   0.2 <tibble [32 x 11]>
#>  4   0.3 <tibble [32 x 11]>
#>  5   0.4 <tibble [32 x 11]>
#>  6   0.5 <tibble [32 x 11]>
#>  7   0.6 <tibble [32 x 11]>
#>  8   0.7 <tibble [32 x 11]>
#>  9   0.8 <tibble [32 x 11]>
#> 10   0.9 <tibble [32 x 11]>
#> 11   1   <tibble [32 x 11]>

^{Created on 2021-10-14 by the reprex package (v2.0.1)}

But if you wanted more complex tuning grids, you'd have to set them up yourself, which doesn't use the full functionality of tune/dials.

[Max]

We've really looked into using tune() within formulas (motivated by tuning the amount of smooth GAM models). Our current thinking is that it is infeasible to do it this way.

However, it is possible using workflow sets. You can pass a list of formulas that have the specific changes that you want to evaluate. For the GAM example:

list(
  disp_10 = mpg ~ s(disp, k = 10) +   wt  +   gear,
  disp_20 = mpg ~ s(disp, k = 20) +   wt  +   gear,
  wt      = mpg ~   disp          + s(wt) +   gear,
  gear    = mpg ~   disp          +   wt  + s(gear)
)

It would be good feedback to see what you and others think about this.

[markjrieke]

I haven't yet worked w/workflowsets, but I'll try giving it a shot sometime soon & let you know how it works out! At first glance, it looks like any more complex space-filling grids for these other "tuning parameters" would be out of reach (or manually configured)

thanks again!