My question is on the topic of selecting an optimal probability cutoff in a binary classification model - specifically, when should this take place? Allow me first to ramble on a bit...
Though a threshold of 0.5 is universally used as a starting point, it may not be the best for every situation. For example, one could be in a situation where it is worth sacrificing a low false-positive rate in order to achieve a higher sensitivity/recall. In such a case, one might lower the probability threshold to, say, 0.45 in order to increase the number of true positives.
This feels like it should be a tunable hyperparameter in the modeling process - sort of similar to how preprocessing steps (e.g., under_ratio() in downsampling) can be tuned.
Now, I realize there is a problem inherent to this kind of tuning for some accuracy metrics. For example, if we are trying to increase the sensitivity, the "best" probability threshold will (pretty much) always be the lowest of the range of values provided, and the opposite being true for precision.
tl;dr
But I guess back to my initial question: when should the probability threshold be evaluated and selected? While fitting resamples? In conjunction with tuning other hyperparameters? After tuning has been done and your final model has been fit? What are the best practices here?
As always, thanks!