Optimizing on partial AUC during model parameter tuning?

Machine Learning and Modeling
1

Rather than using AUC as my optimization parameter during hyper parameter tuning in the tidy models framework, I would like to optimize on partial AUC. I am most interested in differential parameter/tuning performance at very high levels of specificity. I've searched a bit but I'm stuck on how to use/implement such a strategy. Any quick pointers?

2

There is the pROC package, which is very good, where you can use that method (and others).

yardstick used to use pROC but does not anymore since we needed support for case weights and wanted fewer dependencies. If you are using tidymodels, you would have to make your own yardstick metric. We used to have that implemented (maybe in version 0.0.9) so there is a starting point.

3

Thanks, Max. I assume this is reading I should do (Custom metrics • yardstick), and my apologies for not finding it before I posted.