Hi, I trained elastic net and random forest models using caret. My target variable has a moderate class imbalance (16% prevalence), but I chose to not correct the class imbalance as it previously threw off calibration metrics. Instead, I decided to move the classification threshold as an alternative to subsample such as SMOTE, ROSE, and class weights ; all of which severely threw off calibration of the probabilities before.
I have selected an alternative threshold of 0.2 as opposed to the default of 0.5 that gives new classifications, but will the new threshold also affect calibration? If so, how do I get probabilities at the alternative threshold.
Noted- I am measuring calibration based on the curve intercept and slope, Integrated calibration index and overall performance based on brier scores.
Many thanks in advance!