Questions tagged [overfitting]

Modeling error (especially sampling error) instead of replicable and informative relationships among variables improves model fit statistics, but reduces parsimony, and worsens explanatory and predictive validity.

Filter by
Sorted by
Tagged with
45
votes
4answers
17k views

What should I do when my neural network doesn't generalize well?

I'm training a neural network and the training loss decreases, but the validation loss doesn't, or it decreases much less than what I would expect, based on references or experiments with very similar ...
31
votes
2answers
19k views

Does it make sense to combine PCA and LDA?

Assume I have a dataset for a supervised statistical classification task, e.g., via a Bayes' classifier. This dataset consists of 20 features and I want to boil it down to 2 features via ...
112
votes
21answers
36k views

What's a real-world example of “overfitting”?

I kind of understand what "overfitting" means, but I need help as to how to come up with a real-world example that applies to overfitting.
21
votes
2answers
50k views

Dealing with singular fit in mixed models

Let's say we have a model ...
52
votes
6answers
107k views

Random Forest - How to handle overfitting

I have a computer science background but am trying to teach myself data science by solving problems on the internet. I have been working on this problem for the last couple of weeks (approx 900 rows ...
10
votes
1answer
457 views

How to simplify a singular random structure when reported correlations are not near +1/-1

I have read in several answers to questions on this site that the best way to choose the random structure for a mixed effects model is by using theoretical knowledge. On the other hand I have also ...
58
votes
6answers
6k views

Is ridge regression useless in high dimensions ($n \ll p$)? How can OLS fail to overfit?

Consider a good old regression problem with $p$ predictors and sample size $n$. The usual wisdom is that OLS estimator will overfit and will generally be outperformed by the ridge regression estimator:...
18
votes
2answers
12k views

What measure of training error to report for Random Forests?

I'm currently fitting random forests for a classification problem using the randomForest package in R, and am unsure about how to report training error for these ...
34
votes
6answers
33k views

How does cross-validation overcome the overfitting problem?

Why does a cross-validation procedure overcome the problem of overfitting a model?
4
votes
2answers
4k views

Why is my high degree polynomial regression model suddenly unfit for the data?

I'm building a ridge regression model in scikit-learn and trying to find the optimal degree polynomial to use. The data I'm working with is a fairly predictable time series of hourly traffic volumes, ...
22
votes
2answers
20k views

Boosting: why is the learning rate called a regularization parameter?

The learning rate parameter ($\nu \in [0,1]$) in Gradient Boosting shrinks the contribution of each new base model -typically a shallow tree- that is added in the series. It was shown to dramatically ...
6
votes
1answer
3k views

Overfitting on the loss graph, but not the accuracy graph

I am looking at learning curves (CNN for text classification, which is based on this paper) and trying to play with regularization to prevent overfitting. This model uses L2 regularization and dropout....
15
votes
2answers
15k views

Out of Bag Error makes CV unnecessary in Random Forests?

I am fairly new to random forests. In the past, I have always compared the accuracy of fit vs test against fit vs train to detect any overfitting. But I just read here that: "In random forests, ...
31
votes
5answers
7k views

Is an overfitted model necessarily useless?

Assume that a model has 100% accuracy on the training data, but 70% accuracy on the test data. Is the following argument true about this model? It is obvious that this is an overfitted model. The ...
6
votes
3answers
15k views

Random Forest Overfitting R

I used a two-step cforest in my model. the accuracy of the train set is 87%, yet the accuracy of the test set is 57%. This indicates the model is severely overfitting. How to solve this problem? ...
5
votes
4answers
1k views

What exactly is overfitting?

Many people (including me) is thinking or used to think that an overfitted model is the model in which the training error >> the validation error. But after reading this very interesting comment by @...
7
votes
3answers
2k views

Is a lower training accuracy possible in overfitting (one class SVM)

I am using the heart_scale data from LibSVM. The original data includes 13 features, but I only used 2 of them in order to plot the distributions in a figure. Instead of training the binary classifier,...
8
votes
1answer
1k views

Grid fineness and overfitting when tuning $\lambda$ in LASSO, ridge, elastic net

I wonder about the optimal grid fineness and what the relation between grid fineness and overfitting is in regularization methods such as LASSO, ridge regression or elastic net. Suppose I want ...
10
votes
1answer
7k views

How to Identify Overfitting in Convolutional Neural network?

I understand that dropout is used to reduce over fitting in the network. This is a generalization technique. In convolutional neural network how can I identify overfitting? One situation that I can ...
24
votes
1answer
34k views

Discussion about overfit in xgboost

My set-up is the following: I am following the guidlines in "Applied Predictive Modelling". Thus I have filtered correlated features and end up with the following: 4900 data points in the training ...
29
votes
4answers
888 views

Has the journal Science endorsed the Garden of Forking Pathes Analyses?

The idea of adaptive data analysis is that you alter your plan for analyzing the data as you learn more about it. In the case of exploratory data analysis (EDA), this is generally a good idea (you are ...
27
votes
6answers
9k views

Why do smaller weights result in simpler models in regularization?

I completed Andrew Ng's Machine Learning course around a year ago, and am now writing my High School Math exploration on the workings of Logistic Regression and techniques to optimize on performance. ...
10
votes
5answers
7k views

Is using both training and test sets for hyperparameter tuning overfitting?

You have a training and a test set. You combine them and do something like GridSearch to decide the hyperparameters of the model. Then, you fit a model on the training set using these hyperparameters, ...
9
votes
1answer
8k views

Neural network over-fitting

I've learned that over-fitting can be detected by plotting the training error and the testing error versus the epochs. Like in: I've been reading this blogpost where they say the neural network, net5 ...
2
votes
1answer
545 views

Nested nested cross validation for model selection

Suppose I have decided to evaluate the following model selection procedure (let's call it PROC(1) ) START PROCEDURE: For alpha in [0,1,2...1000]: Get the K-fold cross validation error of M(alpha)...
30
votes
0answers
3k views

When wouldn't I use LASSO for model selection? [duplicate]

Assume that you need to build a linear model to make predictions for new observations, and that there is uncertainty about which subset of variables should be included in the model. You are only ...
18
votes
0answers
9k views

Is cross-validation enough to prevent overfitting? [duplicate]

If I have a data, and I run a classification (let's say random forest on this data) with cross validation (let's say 5-folds), could I conclude that there is no over fitting in my method?
2
votes
1answer
3k views

Test overfitting of logistic regression with limited volume

I have a set of samples with two labels red and black. I can build a logistic regression model to predict the label colour. Once a model is built, I would like to test whether it is overfitting or not....
2
votes
1answer
58 views

What could be the reasons that making validation loss jumping up and down?

I am building some image classification model with reasonable size data (~3K) images in both training and validation set. However, I noticed the performance on validation set is not stable. For ...
63
votes
6answers
71k views

How is it possible that validation loss is increasing while validation accuracy is increasing as well

I am training a simple neural network on the CIFAR10 dataset. After some time, validation loss started to increase, whereas validation accuracy is also increasing. The test loss and test accuracy ...
13
votes
3answers
7k views

Bayesian vs MLE, overfitting problem

In Bishop's PRML book, he says that, overfitting is a problem with Maximum Likelihood Estimation (MLE), and Bayesian can avoid it. But I think, overfitting is a problem more about model selection, ...
31
votes
5answers
36k views

Overfitting a logistic regression model

Is it possible to overfit a logistic regression model? I saw a video saying that if my area under the ROC curve is higher than 95%, then its very likely to be over fitted, but is it possible to ...
14
votes
1answer
5k views

Train vs Test Error Gap and its relationship to Overfitting : Reconciling conflicting advice

There seems to be conflicting advice out there about how to handle comparing train vs test error, particularly when there is a gap between the two. There seem to be two schools of thought that to me, ...
13
votes
2answers
4k views

Can one (theoretically) train a neural network with fewer training samples than weights?

First of all: I know, there is no general number of sample size required to train a neural network. It depends on way too many factors like complexity of the task, noise in the data and so on. And the ...
10
votes
1answer
8k views

Techniques to detect overfitting

I had a job interview for a data science position. During the interview, I was asked what do I do to make sure the model is not overfitting. My first answer was to use cross-validation to assess the ...
14
votes
2answers
760 views

Optimization: The root of all evil in statistics?

I have heard the following expression before: "Optimization is the root of all evil in statistics". For example, the top answer in this thread makes that statement in reference to the danger of ...
12
votes
3answers
479 views

Is it better to select distributions based on theory, fit or something else?

This is bordering on a philosophical question, but I am interested in how others with more experience think about distribution selection. In some cases it seems clear that theory might work best (mice ...
9
votes
1answer
4k views

Additive bias in xgboost (and its correction?)

I am taking part in a competition right now. I know it is my job to do that well, but maybe somebody wants to discuss my problem and its solution here as this could be helfull for others in their ...
6
votes
2answers
3k views

Knots in Smoothing Splines

In Introduction to Statistical Learning, there's this line under the section describing Smoothing Spline's tuning parameter $\lambda$: In fitting a smoothing spline, we do not need to select the ...
10
votes
3answers
7k views

Early stopping vs cross validation

I'm currently using early stopping in my work to prevent over fitting. Specifically those taken form Early Stopping But When?. I'm now wanting to compare to other classification algorithms where it ...
9
votes
1answer
542 views

What is cross-validation?

I'm having trouble understanding what cross-validation is. Also, what is the connection between cross-validation and the issue of model overfitting?
4
votes
3answers
2k views

How much is overfitting?

How much is overfitting? For example, results on seen data is between 1 to 15% better than unseen data. Is there a range of value for example 2% where it is considered normal and not overfitting? ...
3
votes
1answer
4k views

How can training and testing error comparisons be indicative of overfitting?

In my research group we are discussing if it is possible to say a model has overfitting just by comparing the two errors, without knowing anything more about the experiment. ps: I am personally ...
1
vote
2answers
2k views

Better accuracy with validation set than test set

I trained a model with some algorithms like random forest, logistic regression and so on. My dataset was split into 80% CV train data (so actually 60% of data to train the model and 20 % for testing ...
6
votes
3answers
139 views

Impossible to overfit when the data generating process is deterministic?

For a stochastic data generating process (DGP) $$ Y=f(X)+\varepsilon $$ and a model producing a point prediction $$ \hat{Y}=\hat{f}(X), $$ the bias-variance decomposition is \begin{align} \text{Err}(...
4
votes
1answer
417 views

Cannot overfit on the IRIS dataset

I am playing with the IRIS dataset and want to see underfitting and overfitting in action. I am using a multilayer perceptron (2 layers). It is pretty easy to underfit (see the plot below), but I am ...
0
votes
3answers
87 views

Is this a case of network overfitting?

I am writing a network that classifies different species of butterfly, I have 9 epochs total. I have reached a wall as my major is in Physics, I am wondering if anyone can spot any distinct issues ...
14
votes
3answers
18k views

How to detect when a regression model is over-fit?

When you are the one doing the work, being aware of what you are doing you develop a sense of when you have over-fit the model. For one thing, you can track the trend or deterioration in the Adjusted ...
27
votes
2answers
5k views

Is it true that Bayesian methods don't overfit?

Is it true that Bayesian methods don't overfit? (I saw some papers and tutorials making this claim) For example, if we apply a Gaussian Process to MNIST (handwritten digit classification), but only ...
14
votes
2answers
7k views

How does k-fold cross validation fit in the context of training/validation/testing sets?

My main question is with regards trying to understand how k-fold cross-validation fits in the context of having training/validation/testing sets (if it fits at all in such context). Usually, people ...