roc_auc() for each class in a multiclass classification task

[vrsteiger]

Hi all,

In order to gain more insights into the results of an analysis.
I was wondering if there is a way to get a roc_auc() value for each class in a classification task with three classes.

my analysis consisted of a workflow_map using three models and two preprocessing steps. I've used last_fit on the best model regarding to roc_auc(). collect_predictions(final_res) will give me the predictions for the test set (test_predictions)

Now I'm not sure if its correct to use:

test_predictions %>% group_by(.pred_class) %>% roc_auc(truth_col, c("class1", "class2", "class3"))

in order to get the roc_auc score for each class separately.

--> 9 Judging Model Effectiveness | Tidy Modeling with R

Example:

library(tidymodels)
library(palmerpenguins)

penguins_split <- initial_split(penguins,
                                strata = "species")
penguins_train <- training(penguins_split)

rf_spec <- rand_forest() |>
  set_mode("classification")

results <- workflow(preprocessor = recipe(species ~ island + year,
                                          data = penguins_train),
                    spec = rf_spec) |>
  last_fit(penguins_split)

results |>
  collect_predictions() |>
  roc_auc(
    truth = species ,
    .pred_Adelie,
    .pred_Chinstrap,
    .pred_Gentoo,
    estimator = "macro_weighted"
  )

And finally:

results |>
  collect_predictions() |>
  group_by(.pred_class) |>
  roc_auc(
    truth = species ,
    .pred_Adelie,
    .pred_Chinstrap,
    .pred_Gentoo,
    estimator = "macro_weighted"
  )