Using proper scoring rule to determine class membership from logistic regression

Question

I am using logistic regression to predict likelihood of an event occurring. Ultimately, these probabilities are put into a production environment, where we focus as much as possible on hitting our "Yes" predictions. It is therefore useful for us to have an idea of what definitive "hits" or "non-hits" might be a priori (before running in production), in addition to other measures we use for informing this determination.

My question is, what would be the proper way to predict a definitive class (1,0) based on the predicted probability? Specifically, I use R's glmnet package for my modeling. This package arbitrarily picks .5 probability as threshold for a yes or no. I believe that I need to take the results of a proper scoring rule, based on predicted probabilities, to extrapolate to a definitive class. An example of my modeling process is below:

mods <- c('glmnet', 'scoring')
lapply(mods, require, character.only = T)

# run cross-validated LASSO regression
fit <- cv.glmnet(x = df1[, c(2:100)]), y = df1[, 1], family = 'binomial', 
type.measure = 'auc')

# generate predicted probabilities across new data
df2$prob <- predict(fit, type="response", newx = df2[, c(2:100)], s = 'lambda.min')

# calculate Brier score for each record
df2$propscore <- brierscore(df2[,1] ~ df2$prob, data = df2)

So I now have a series of Brier scores for each prediction, but then how do I use the Brier score to appropriately weight each likelihood being a yes or no?

I understand that there are other methods to make this determination as well, such as Random Forest.

Answer 1

What you need to do is "put the model into production", that is, as you say, to actually predict which cases will be "yes", what you need is a loss function. You have two possible errors, saying "yes" when no, saying "no" when yes. Do these errors have economical consequences, costs associated? Do these costs depend on other things too, such as some specific covariate value? Then you build these information into a cost function, and you can determine cutoffs by minimizing expected cost. That is the principled solution.