7
$\begingroup$

I am using logistic regression to predict likelihood of an event occurring. Ultimately, these probabilities are put into a production environment, where we focus as much as possible on hitting our "Yes" predictions. It is therefore useful for us to have an idea of what definitive "hits" or "non-hits" might be a priori (before running in production), in addition to other measures we use for informing this determination.

My question is, what would be the proper way to predict a definitive class (1,0) based on the predicted probability? Specifically, I use R's glmnet package for my modeling. This package arbitrarily picks .5 probability as threshold for a yes or no. I believe that I need to take the results of a proper scoring rule, based on predicted probabilities, to extrapolate to a definitive class. An example of my modeling process is below:

mods <- c('glmnet', 'scoring')
lapply(mods, require, character.only = T)

# run cross-validated LASSO regression
fit <- cv.glmnet(x = df1[, c(2:100)]), y = df1[, 1], family = 'binomial', 
type.measure = 'auc')

# generate predicted probabilities across new data
df2$prob <- predict(fit, type="response", newx = df2[, c(2:100)], s = 'lambda.min')

# calculate Brier score for each record
df2$propscore <- brierscore(df2[,1] ~ df2$prob, data = df2)

So I now have a series of Brier scores for each prediction, but then how do I use the Brier score to appropriately weight each likelihood being a yes or no?

I understand that there are other methods to make this determination as well, such as Random Forest.

$\endgroup$
5
$\begingroup$

What you need to do is "put the model into production", that is, as you say, to actually predict which cases will be "yes", what you need is a loss function. You have two possible errors, saying "yes" when no, saying "no" when yes. Do these errors have economical consequences, costs associated? Do these costs depend on other things too, such as some specific covariate value? Then you build these information into a cost function, and you can determine cutoffs by minimizing expected cost. That is the principled solution.

$\endgroup$
  • 1
    $\begingroup$ Yes, this is actually what we need, e.g. cost to review per unit. I will do my homework on creating a proper loss function, but do I include a measure like the scoring rule into this? $\endgroup$ – NiuBiBang Jul 29 '14 at 19:10
  • $\begingroup$ A proper soring rul an a loss function are two distinct concepts! The loss function should only measure the economic consequences, and a proper scoring rule has nothing to do with that. $\endgroup$ – kjetil b halvorsen Jul 30 '14 at 10:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.