1
$\begingroup$

I am looking at regularized logistic regression, (l1 and l2 at the moment) and regularized discriminant analysis.

How do I compare the two? I was thinking of doing gcv on both methods over a set of values of lambdas for each of the three cases (l1, l2, and rda) then choosing the model with the least amount of error, such as AUC or some classification error. Frank Harrell's comments in this post has me confused if this is even a way of doing such a thing: How do we generate the ROC curve for Linear Discriminant Analysis method

Any help or resources would be greatly appreciated.

Improve this question
$\endgroup$
0

2 Answers 2

Reset to default
2
$\begingroup$

I don't think ROC curves help. See this paper for a comparison of unpenalized logistic models and LDA.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Appreciate the link. If I understand correctly from your paper, ROC curves only measure the ability of a model to classify what group the data belongs to, but it's lack of ability to tell how reliable a model is, therefore it is not a good way of measuring model performance. $\endgroup$
    – nootodis
    Commented Jan 23, 2016 at 19:06
  • 1
    $\begingroup$ IMHO ROC curves tell you nothing useful of any sort. And I don't want to know about 'ability to classify'; I want to know about ability to predict, and I don't want to use a method such as ROC that invites analysts to use thresholds. For model performance I want to use a proper accuracy score plus make a high resolution nonparametric calibration curve. I do use ROC areas ($c$-index; concordance probability) because it is an interpretable (if insensitive) measure of pure predictive discrimination. $\endgroup$
    – Frank Harrell
    Commented Jan 23, 2016 at 19:24
  • 1
    $\begingroup$ When you say that you "use" an ROC curve, is it for understanding the model's discriminative power after a model is chosen? Not to be used to make any type of model calibration. I see it used a lot for choosing a better threshold value, but from what I read from you, it is better to keep the model in context of it's probability then assigning it a threshold. $\endgroup$
    – nootodis
    Commented Jan 24, 2016 at 14:41
  • 2
    $\begingroup$ Well put. My point is that to get an interpretable (if insensitive) measure of predictive discrimination the curve doesn't help at all, but the area under the curve ($c$-index), which can be computed quickly and easily using rank correlation/Wilcoxon test ideas, does help. It is unfortunate that an idea so clean as concordance happens to coincide with the area under such a silly curve. $\endgroup$
    – Frank Harrell
    Commented Jan 24, 2016 at 19:18
1
$\begingroup$

Here's a totally different criterion I use to choose between (linear) discriminant analysis and logistic regression:

Discriminant analysis primarily models your class as elliptic (Gaussian), whereas logistic regression concentrates more on the class boundaries (SVM do so even more). Cases far in the back will influence discriminant analysis model but not logistic regression. I ask myself:

  • Will cases at the backside of the classes occur (think about data generation process), and
  • should they influence the decision boundary or not?
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.