Overfitting a logistic regression model - Cross Validated most recent 30 from stats.stackexchange.com 2019-07-21T03:05:29Z https://stats.stackexchange.com/feeds/question/71946 http://www.creativecommons.org/licenses/by-sa/3.0/rdf https://stats.stackexchange.com/q/71946 25 Overfitting a logistic regression model carlosedubarreto https://stats.stackexchange.com/users/31090 2013-10-04T22:57:42Z 2016-03-29T15:18:14Z <p>Is it possible to overfit a logistic regression model? I saw a video saying that if my area under the ROC curve is higher than 95%, then its very likely to be over fitted, but is it possible to overfit a logistic regression model?</p> https://stats.stackexchange.com/questions/71946/-/71950#71950 29 Answer by AdamO for Overfitting a logistic regression model AdamO https://stats.stackexchange.com/users/8013 2013-10-04T23:48:38Z 2013-10-07T18:48:43Z <p>Yes, you can overfit logistic regression models. But first, I'd like to address the point about the AUC (Area Under the Receiver Operating Characteristic Curve): There are no universal rules of thumb with the AUC, ever ever ever. </p> <p>What the AUC <em>is</em> is the probability that a randomly sampled positive (or case) will have a higher marker value than a negative (or control) because the AUC is mathematically equivalent to the U statistic.</p> <p>What the AUC <em>is not</em> is a standardized measure of predictive accuracy. Highly deterministic events can have single predictor AUCs of 95% or higher (such as in controlled mechatronics, robotics, or optics), some complex multivariable logistic risk prediction models have AUCs of 64% or lower such as breast cancer risk prediction, and those are respectably high levels of predictive accuracy. </p> <p>A sensible AUC value, as with a power analysis, is prespecified by gathering knowledge of the background and aims of a study <em>apriori</em>. The doctor/engineer describes what they want, and you, the statistician, resolve on a target AUC value for your predictive model. <em>Then</em> begins the investigation.</p> <p>It is indeed possible to overfit a logistic regression model. Aside from linear dependence (if the model matrix is of deficient rank), you can also have perfect concordance, or that is the plot of fitted values against Y perfectly discriminates cases and controls. In that case, your parameters have not converged but simply reside somewhere on the boundary space that gives a likelihood of $\infty$. Sometimes, however, the AUC is 1 by random chance alone.</p> <p>There's another type of bias that arises from adding too many predictors to the model, and that's small sample bias. In general, the log odds ratios of a logistic regression model tend toward a biased factor of $2\beta$ because of non-collapsibility of the odds ratio and zero cell counts. In inference, this is handled using conditional logistic regression to control for confounding and precision variables in stratified analyses. However, in prediction, you're SooL. There is no generalizable prediction when you have $p \gg n \pi(1-\pi)$, ($\pi = \mbox{Prob}(Y=1)$) because you're guaranteed to have modeled the "data" and not the "trend" at that point. High dimensional (large $p$) prediction of binary outcomes is better done with machine learning methods. Understanding linear discriminant analysis, partial least squares, nearest neighbor prediction, boosting, and random forests would be a very good place to start.</p> https://stats.stackexchange.com/questions/71946/-/156886#156886 4 Answer by curious_cat for Overfitting a logistic regression model curious_cat https://stats.stackexchange.com/users/21072 2015-06-14T07:34:51Z 2015-06-14T07:34:51Z <p>Is there any model, leave aside logistic regression, that it is not possible to overfit? </p> <p>Overfitting arises fundamentally because you fit to a sample &amp; not the whole population. Artifacts of your sample can seem like features of the population and they are not and hence overfitting hurts. </p> <p>It is akin to a question of external validity. Using only the sample you are trying to get a model that gives you the best performance on the real population which you cannot see. </p> <p>Sure, some model forms or procedures are more likely to overfit than others but no model is ever truly immune from overfitting, is it? </p> <p>Even out-of-sample validation, regularization procedures etc. can only guard against over-fitting but there's no silver bullet. In fact, if one were to estimate one's confidence in making a real world prediction based on a fitted model one must always assume that some degree of overfitting has indeed happened. </p> <p>To what extent might vary, but even a model validated on a hold out dataset will rarely yield in-wild performance that matches what was obtained on the hold-out dataset. And overfitting is a big causative factor. </p> https://stats.stackexchange.com/questions/71946/-/156912#156912 5 Answer by Frank Harrell for Overfitting a logistic regression model Frank Harrell https://stats.stackexchange.com/users/4253 2015-06-14T12:12:18Z 2015-06-14T12:12:18Z <p>You can overfit with any method, even if you fit the whole population (if the population is finite). There are two general solutions to the problem: (1) penalized maximum likelihood estimation (ridge regression, elastic net, lasso, etc.) and (2) the use of informative priors with a Bayesian model.</p> <p>When $Y$ has limited information (e.g., is binary or is categorical but unordered), overfitting is more severe just because whenever you have low information it is like having a smaller sample size. For example a sample of size 100 from a continuous $Y$ may have the same information as a sample of size 250 from a binary $Y$, for the purposes of statistical power, precision, and overfitting. Binary $Y$ assumes an all-or-nothing phenomenon and has 1 bit of information. Many continuous variables have at least 5 bits of information.</p> https://stats.stackexchange.com/questions/71946/-/182748#182748 0 Answer by María Frances Gaska for Overfitting a logistic regression model María Frances Gaska https://stats.stackexchange.com/users/95736 2015-11-20T12:59:48Z 2015-11-20T12:59:48Z <p>What we do with the Roc to check for overfitting is to separete the dataset randomly in training and valudation and compare the AUC between those groups. If the AUC is "much" (there is also no rule of thumb) bigger in training then there might be overfitting. </p> https://stats.stackexchange.com/questions/71946/-/204368#204368 5 Answer by user110267 for Overfitting a logistic regression model user110267 https://stats.stackexchange.com/users/110267 2016-03-29T15:18:14Z 2016-03-29T15:18:14Z <p>In simple words.... an overfitted logistic regression model has large variance, means decision boundry changes largely for small change in variable magnitude. consider following image the right most one is overfitted logistic model, its decision boundry has large no. of ups and downs while the middel one is just fit it has moderate variance and moderate bias. the left one is underfit it has high bias but very less variance. one more thing_ An overfitted regrresion model have too many features while underfit model has very less no. of features. <a href="https://i.stack.imgur.com/20hsn.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/20hsn.jpg" alt="image showing underfit, justfit and overfit logistic resion model. Blue colured marker is showing decision boundry."></a></p>