Hello, I'm trying to evaluate graphically a workflow_set after a tuning process and I've seen a lot of publications using caret's bwplot() function, which I think are quite nice. I was surprised I couldn't find anything similar on tidymodels.
Ideally, I would like to compare in a boxplot the metrics for each of the best set of hyperparameters from each model that has been tuned.
I have achieved the results after cherry-picking the model's metrics from the workflow_set tuning results, but this is tedious and very prone to errors.
nnet <- models_tuned %>%
unnest(result) %>%
unnest(.metrics) %>%
filter(wflow_id=="norm_nnet",
.iter==0,
.config=="Preprocessor1_Model02",
.metric %in% c("roc_auc", "f_meas", "sens", "spec")
) %>%
select(wflow_id, id, id2, .metric, .estimate)
rfor <- models_tuned %>%
unnest(result) %>%
unnest(.metrics) %>%
filter(wflow_id=="norm_rfor",
.iter==2,
.metric %in% c("roc_auc", "f_meas", "sens", "spec")
) %>%
select(wflow_id, id, id2, .metric, .estimate)
knn <- models_tuned %>%
unnest(result) %>%
unnest(.metrics) %>%
filter(wflow_id=="norm_knn",
.iter==0,
.config=="Preprocessor1_Model02",
.metric %in% c("roc_auc", "f_meas", "sens", "spec")
) %>%
select(wflow_id, id, id2, .metric, .estimate)
glm <- models_tuned %>%
unnest(result) %>%
unnest(.metrics) %>%
filter(wflow_id=="norm_glm",
.config=="Preprocessor1_Model01",
.metric %in% c("roc_auc", "f_meas", "sens", "spec")
) %>%
select(wflow_id, id, id2, .metric, .estimate)
svm <- models_tuned %>%
unnest(result) %>%
unnest(.metrics) %>%
filter(wflow_id=="norm_svm",
.iter==2,
.metric %in% c("roc_auc", "f_meas", "sens", "spec")
) %>%
select(wflow_id, id, id2, .metric, .estimate)
clean <- bind_rows(nnet, rfor, knn, glm, svm)
clean %>%
group_by(wflow_id, .metric) %>%
summarise(mean=mean(.estimate, na.rm=T), n=n())
ggplot(clean, aes(.estimate, wflow_id)) +
geom_boxplot(na.rm=T) +
facet_wrap(~.metric)
Is there a better way to do this? I know I can get some graphs with autoplot(), but they don't give that much information.
Thanks a lot!