I am using logistic regression to predict likelihood of an event occurring. Ultimately, these probabilities are put into a production environment, where we focus as much as possible on hitting our "Yes" predictions. It is therefore useful for us to have an idea of what definitive "hits" or "non-hits" might be a priori (before running in production), in addition to other measures we use for informing this determination.
My question is, what would be the proper way to predict a definitive class (1,0) based on the predicted probability? Specifically, I use R's
glmnet package for my modeling. This package arbitrarily picks .5 probability as threshold for a yes or no. I believe that I need to take the results of a proper scoring rule, based on predicted probabilities, to extrapolate to a definitive class. An example of my modeling process is below:
mods <- c('glmnet', 'scoring') lapply(mods, require, character.only = T) # run cross-validated LASSO regression fit <- cv.glmnet(x = df1[, c(2:100)]), y = df1[, 1], family = 'binomial', type.measure = 'auc') # generate predicted probabilities across new data df2$prob <- predict(fit, type="response", newx = df2[, c(2:100)], s = 'lambda.min') # calculate Brier score for each record df2$propscore <- brierscore(df2[,1] ~ df2$prob, data = df2)
So I now have a series of Brier scores for each prediction, but then how do I use the Brier score to appropriately weight each likelihood being a yes or no?
I understand that there are other methods to make this determination as well, such as Random Forest.