1
$\begingroup$

I would like to use a logistic regression for estimating the parameters of the logit function by using the maximum likelihood estimate. This amounts to minimizing the log-loss function, instead of minimizing another loss function such as squared error.

To assess a model's predictive power I want to employ this strategy to the training set. This is essentially minimizing the log-loss cost function with respect to the parameters of the model. But now I can use a different loss function to assess predictive performance such as briers score? Or log-loss as was used in the parameter estimation? Or it doesn't matter since these are two two loss functions are answer different questions of model fitting and performance checking?

It would seem I want to find the model's parameters in which it finds the minimum of a loss function, then the same loss function is used to assess predictive performance of the model.

Source: Defining predictive vs estimation beginning of intro https://www.ine.pt/revstat/pdf/rs070102.pdf

Improve this question
$\endgroup$
1
$\begingroup$

The gold standard is what maximum likelihood maximizes (logarithmic probability accuracy scoring rule). But it is often a good idea to use another proper accuracy score that was not optimized by the model fitting. In this case Brier score would be an excellent choice, and it does not go to infinity if you make one total mistake (prob=0 for Y=1 or prob=1 for Y=0). And don't forget to look at calibration if absolute accuracy is an issue.

Improve this answer
$\endgroup$
  • $\begingroup$ I decide upon a model. I run a gcv that fits the model for every gcv run by using MLE. Then I keep track of the predicted accuracy using loss functions like log-loss, briers, and whatever else since they have their own strengths. Then when that model is chosen based upon my decision based on the prediction error, I run the model on the entire dataset and do a calibration plot to make sure the fit is proper. $\endgroup$ – nootodis Feb 4 '16 at 20:59
  • $\begingroup$ The last step is biased. I suggest you correct for the bias using the Efrong-Gong optimism bootstrap (see R rms package). $\endgroup$ – Frank Harrell Feb 4 '16 at 21:22
  • $\begingroup$ I am comparing penalized likelihood models with unpenalized, so that would take place of bootstrap you suggested? I will bring bootstrap in later but just trying to understand everything carefully. I was thinking calibration plot like a qq-plot to assess linear model parameter fit, which seems like should be done as the final step to ensure the model chosen for prediction has a good parameter fit. $\endgroup$ – nootodis Feb 4 '16 at 21:32
  • $\begingroup$ loess is a good way to get a nonparametric calibration curve. Bootstrapping can be done with or without penalization; that is a separate issue. $\endgroup$ – Frank Harrell Feb 4 '16 at 22:50
  • $\begingroup$ Thanks for holding my hand through this. The calibration curve is just to see that the model fits its parameters correctly. If the model has bad "calibration", then I would choose the model that has the second best predictive ability then check it's calibration curve, and so on. So the calibration curve is done as the last step of modelling after we have chosen a model that has the best predictive ability. $\endgroup$ – nootodis Feb 4 '16 at 22:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.