Choosing between logistic and discriminant

I am looking at regularized logistic regression, (l1 and l2 at the moment) and regularized discriminant analysis.

How do I compare the two? I was thinking of doing gcv on both methods over a set of values of lambdas for each of the three cases (l1, l2, and rda) then choosing the model with the least amount of error, such as AUC or some classification error. Frank Harrell's comments in this post has me confused if this is even a way of doing such a thing: How do we generate the ROC curve for Linear Discriminant Analysis method

Any help or resources would be greatly appreciated.

I don't think ROC curves help. See this paper for a comparison of unpenalized logistic models and LDA.

• Appreciate the link. If I understand correctly from your paper, ROC curves only measure the ability of a model to classify what group the data belongs to, but it's lack of ability to tell how reliable a model is, therefore it is not a good way of measuring model performance. – nootodis Jan 23 '16 at 19:06
• IMHO ROC curves tell you nothing useful of any sort. And I don't want to know about 'ability to classify'; I want to know about ability to predict, and I don't want to use a method such as ROC that invites analysts to use thresholds. For model performance I want to use a proper accuracy score plus make a high resolution nonparametric calibration curve. I do use ROC areas ($c$-index; concordance probability) because it is an interpretable (if insensitive) measure of pure predictive discrimination. – Frank Harrell Jan 23 '16 at 19:24
• When you say that you "use" an ROC curve, is it for understanding the model's discriminative power after a model is chosen? Not to be used to make any type of model calibration. I see it used a lot for choosing a better threshold value, but from what I read from you, it is better to keep the model in context of it's probability then assigning it a threshold. – nootodis Jan 24 '16 at 14:41
• Well put. My point is that to get an interpretable (if insensitive) measure of predictive discrimination the curve doesn't help at all, but the area under the curve ($c$-index), which can be computed quickly and easily using rank correlation/Wilcoxon test ideas, does help. It is unfortunate that an idea so clean as concordance happens to coincide with the area under such a silly curve. – Frank Harrell Jan 24 '16 at 19:18

Here's a totally different criterion I use to choose between (linear) discriminant analysis and logistic regression:

Discriminant analysis primarily models your class as elliptic (Gaussian), whereas logistic regression concentrates more on the class boundaries (SVM do so even more). Cases far in the back will influence discriminant analysis model but not logistic regression. I ask myself:

• Will cases at the backside of the classes occur (think about data generation process), and
• should they influence the decision boundary or not?