Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Is it possible to overfit a logistic regression model? I saw a video saying that if my area under the ROC curve is higher than 95%, then its very likely to be over fitted, but is it possible to overfit a logistic regression model?

share|improve this question
    
Can you say which video, or at least give a little more context? –  Glen_b Oct 12 '13 at 5:43
2  
Sure @Glen_b, the video was this: link The comment was at 40min. It was saying that thing: when ROC have the AUC between 0,5 and 0,6 it was Poor. If between 0,6 and 0,7 it´s below average. If between 0,7 and 0,75 it´s a average/Good. It betwwen 0,75 and 0,8 it´s good. If between 0,8 and 0,9 its Excelent. If higher than 0,9 it´s suspicious and if higher then 0,95 it´s overfitted. I´ve found that explanation very easy to understand, but it´s right? Because I´m searching for something to backup that thought but i´m not finding. –  carlosedubarreto Oct 12 '13 at 10:35
    
And Glen_B, the explanation that @AdamO gave seems that the explanation that I saw on video wasn´t exactly right, but maybe I´ve misanderstood Adam´s explanation. These Statiscs stuff is very complex, but It´s a real pleasure to dig deep into it. :) –  carlosedubarreto Oct 12 '13 at 10:38
    
I think AdamO's explanation is good (I upvoted it), but questions are supposed to be permanent resources; a later reader (e.g. someone with a similar question) may want the context of knowing what was said. I think your description in the comment gives enough context for most people and the link will do for the rest. So thank you! You've made your question more useful. –  Glen_b Oct 12 '13 at 10:43
    
Many thanks @Glen_b, I´m learning how to use this awesome tool we have(this forum). I´ll keep your tip in mind when asking new questions. –  carlosedubarreto Oct 12 '13 at 10:47

1 Answer 1

up vote 17 down vote accepted

Yes, you can overfit logistic regression models. But first, I'd like to address the point about the AUC (Area Under the Receiver Operating Characteristic Curve): There are no universal rules of thumb with the AUC, ever ever ever.

What the AUC is is the probability that a randomly sampled positive (or case) will have a higher marker value than a negative (or control) because the AUC is mathematically equivalent to the U statistic.

What the AUC is not is a standardized measure of predictive accuracy. Highly deterministic events can have single predictor AUCs of 95% or higher (such as in controlled mechatronics, robotics, or optics), some complex multivariable logistic risk prediction models have AUCs of 64% or lower such as breast cancer risk prediction, and those are respectably high levels of predictive accuracy.

A sensible AUC value, as with a power analysis, is prespecified by gathering knowledge of the background and aims of a study apriori. The doctor/engineer describes what they want, and you, the statistician, resolve on a target AUC value for your predictive model. Then begins the investigation.

It is indeed possible to overfit a logistic regression model. Aside from linear dependence (if the model matrix is of deficient rank), you can also have perfect concordance, or that is the plot of fitted values against Y perfectly discriminates cases and controls. In that case, your parameters have not converged but simply reside somewhere on the boundary space that gives a likelihood of $\infty$. Sometimes, however, the AUC is 1 by random chance alone.

There's another type of bias that arises from adding too many predictors to the model, and that's small sample bias. In general, the log odds ratios of a logistic regression model tend toward a biased factor of $2\beta$ because of non-collapsibility of the odds ratio and zero cell counts. In inference, this is handled using conditional logistic regression to control for confounding and precision variables in stratified analyses. However, in prediction, you're SooL. There is no generalizable prediction when you have $p \gg n \pi(1-\pi)$, ($\pi = \mbox{Prob}(Y=1)$) because you're guaranteed to have modeled the "data" and not the "trend" at that point. High dimensional (large $p$) prediction of binary outcomes is better done with machine learning methods. Understanding linear discriminant analysis, partial least squares, nearest neighbor prediction, boosting, and random forests would be a very good place to start.

share|improve this answer
    
When you say $y$, I guess you mean the proportion of the time that y=1? And not the sum of the number of times that y=1? –  ACD Oct 7 '13 at 18:35
    
That was confusing, p is the number of parameters in the model, I now use $\pi$ for the proportion. Thanks for pointing that out. –  AdamO Oct 7 '13 at 18:49
    
How do you determine the appropriate AUC value to aim for? –  Kevin H. Lin Dec 9 at 3:20
    
@KevinH.Lin It depends on the nature of the question. The more in which you incorporate contextually appropriate knowledge, the better. This would be the underlying prevalence or burden of disease or condition which the model assesses, performance of existing (competing) models, cost-effectiveness tradeoffs, and policies surrounding the adoption of new practices and/or recommendations. Nothing about it is black and white, but like so many things, you need to argue compellingly to convince and reason in favor of an AUC value that you, as the statistician, prespecify. –  AdamO Dec 9 at 19:37
    
@AdamO OK, let me ask in another way: Where can I read more about how to pick an appropriate AUC for different types of problems? Or where can I read about some different examples and the types of arguments that people make? –  Kevin H. Lin Dec 9 at 20:48

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.