I'm reading the Caret documentation here about adding alternate performance metrics.
my_summary <- function(data, lev = NULL, model = NULL){
a1 <- defaultSummary(data, lev, model)
b1 <- twoClassSummary(data, lev, model)
c1 <- prSummary(data, lev, model)
d1 <- MLmetrics::PRAUC(y_pred = data[, "pred"], y_true = data[, "obs"])
out <- c(a1, b1, c1, d1)
out}
Then in trainControl:
train_control <- trainControl(
method = "cv",
number = 5,
classProbs = TRUE,
verboseIter = TRUE,
savePredictions = TRUE,
allowParallel = TRUE
)
If I exclude d1, everything works great. However, when I try to use MLmetrics::PRAUC()
I get an error "Error in { : task 1 failed - "Format of predictions is invalid."
How can I add PR AUC as a custom function to Caret, either using MLmetrics or not?
Rest of XGB via Caret code:
## tuning grid
tune_grid <- expand.grid(nrounds = 200,
max_depth = 5,
eta = 0.05,
gamma = 0.01,
colsample_bytree = 0.75,
min_child_weight = 0,
subsample = 0.5)
## xgb
### pre process does not work with formula interface, only x, y
XGBs <- foreach(i = targets) %do% {
target <- as.factor(median_training_data[i][[1]]) %>% make.names()
xgb <- train(x = select(median_training_data, -i),
y = target,
method = "xgbTree",
trControl = train_control,
tuneGrid = tune_grid,
tuneLength = 10)
saveRDS(xgb, file = paste0(i, ".rds"))
return(1) # make loop not store in memory
}