Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Questions tagged [overfitting]

Modeling error (especially sampling error) instead of replicable and informative relationships among variables improves model fit statistics, but reduces parsimony, and worsens explanatory and predictive validity.

Filter by
Sorted by
Tagged with
1
vote
2answers
33 views

Linear model selection after bootstrapping without overfitting

I am trying to develop a model between a 19 year record of climate data and a 19 year record of ice-off dates on rivers. The two variables are linearly correlated. The goal is to build a linear model ...
4
votes
1answer
219 views

Is it ever possible for a model to attain 100% accuracy on both train and test and has not overifitted?

I'm curious to know whether we have such a case and if they exist how good/bad it is! I'm coming from this question and I'm thinking whether overfitting is an unpreventable concept that occurs ...
1
vote
2answers
28 views

Does a CNN fully memorize ground truth if it has more parameters than training pixels?

ResNet consists of 25M trainable parameters. If only 30% of 600 $512 \times 512$ images is annotated, there are $600 * 512 * 512 * ~0.3 = 47,185,920$ ground truth pixels. A parameter is a floating ...
1
vote
1answer
26 views

overfitting and selection model

Currently I’m focusing on model selection criteria, more specifically: sequential hypothesis testing, information criteria (like BIC and AIC), Lasso. All of these in regression framework. These ...
0
votes
0answers
17 views

How can underfitting and overfitting Classifiers of the same Type perform equally worse?

An rbf support vector classifier is instantiated with different values for C and gamma. Then learning curves are plotted for each of the classifiers. Some classifiers consistently achieve very high ...
1
vote
1answer
23 views

modelling on differenced data

I have a time-series data that I want to model using machine learning models like Lasso Regression, Ridge, elastic net, etc. However, in order to make it stationary, I difference the output variable, ...
0
votes
2answers
47 views

Is it possible to define an optimal fit?

Let assume that we have n pairs of real numbers: (x_1, y_1), (x_2, y_2), ..., (x_n, y_n) Let as also assume that ...
1
vote
1answer
39 views

Cross-validation for (hyper)parameter tuning to be performed in validation set or training set?

I am learning about the use of cross-validation with grid-search to choose the best hyperparameter for SVM. The problem I came across is the references and examples of its application do not follow a ...
0
votes
2answers
45 views

Is it possible to know if a machine learning model is overfitted from negative samples?

I have a trained model and with this I can classify faces, if I test the classifier by entering the same negative samples (not faces) with which I train, is it possible to know if my model is ...
0
votes
1answer
45 views

Is increasing the class weight of minority class in Random Forest algorithm decreasing the performance?

I am trying to classify an imbalanced dataset (census dataset with approx. 3:1 imbalance) using Random Forest algorithm in python, and what I observed that increasing the class weight of the minority ...
0
votes
2answers
31 views

Does over fitting a model affect R Squared only or adjusted R Squared too?

The total sum of squares for the variable being predicted is as the following: $$\mathrm{TSS}=\sum_{i=1}^{n}\left(y_{i}-\bar{y}\right)^2$$ and the residual sum of squares from the predictions from ...
2
votes
1answer
28 views

Overfitting in recommender systems

So I want to know whether or not my models are overfitting or the difference between train and validation errors are decent. $L$: is the number of neighbors The first column is the train error ...
4
votes
0answers
21 views

What are the most common techniqes for preventing overfitting? [duplicate]

How is this issue dealt with, in practice?
0
votes
0answers
15 views

Understading Overfitting from Precision and Recall scores

Can I understand if my model is overfitting or underfitting from its precision and recall scores that it has on training and test datasets?
2
votes
1answer
31 views

Fitting data with unknown noise with complicated model — can anything be done?

I have some experimental data, where due to the nature of the experiment (which I am not familiar with) the errors are not known. The data follow a rough non-linear trend but clearly are noisy as for ...
0
votes
0answers
17 views

Using hold-out method for validation set: How to choose a DL model with model selection?

After >170 deep learning experiments were I did a (almost) full factorial design with >15 factors. I cannot measure performance with cross validation because that would require to much training of ...
0
votes
0answers
10 views

testing results better than validation in nested cross-validation

I am implementing a code for applying nested cross validation and as you know the inner loop is for hyper parameter tuning and the outer loop is for testing the general performance: In each fold of ...
1
vote
1answer
30 views

SEM output with a DWLS est - non normal distributed variables: feedback, possible overfitting?

I have a sem model with non-normal distributed indicators measured with Likert-scale (7 points - agreement). I have 4 LVs and 1 observed (N=287, data collection still in progress). As variables in my ...
1
vote
0answers
20 views

What are some reasons that a regression perfectly fits a test set? [closed]

I recently built a simple linear model that I trained using a standard 30-70 split on my data set. To my surprise, when I tested my model on unseen data, it reported the following: With a linear ...
0
votes
1answer
63 views

Can a relationship between x and y be modeled, if all the data points fill the area under a curve?

I'd like to derive an equation that enables me to calculate y based on x. I'm having troubles figuring out how to do this as my data doesn't form a line/curve, but rather creates an edge (image below)...
2
votes
2answers
41 views

Overfitting refers specifically to the case in which a less flexible model would have yielded a smaller test MSE

This line is written in ISLR : Overfitting refers specifically to the case in which a less flexible model would have yielded a smaller test MSE. I am unable to get this line, can anyone explain to me ...
5
votes
1answer
130 views

Peeking Inside the Black Box, can Feature importance indicate overfit?

my basic question is: can permutation feature importance be used to identify overfitting? when you have a binary classifcation problem with balanced classes (i.e. 70 x yes, 70 x no), when none of the ...
1
vote
1answer
50 views

Overfitting model or issue of categorical predictors?

Is it possible to overfit a model by virtue of having too many categorical variables? I have 3 categorical variables and my dependent measure is continuous (or a ratio I guess, I'm measuring ...
0
votes
2answers
64 views

Statistical evidence that the AUC was not overfitted to the model. With N=119, C-stat = 0.81 seems optimistic. Optimism-adjusted?

My data have 119 cases and we did ROC for x (continuous variable) to predict postoperative y (categorical variable) available here, we got a comment from a reviewer asking: Please provide ...
0
votes
1answer
16 views

General Advice - Neural Network Optimization for Noisy Label Training

I'm new to Neural Networks. Trying to get some general advice. Multi Class, 3 classes Has noisy labels, with somewhere between 60 and 80 percent accuracy Huge amount of training with the issues ...
1
vote
1answer
25 views

How to identify a case of overfitting using stratified k fold cross validation?

I am working on a problem with a very small amount of data of 211 examples. The problem is a binary classification problem with 2 sets of classes. The data is highly imbalanced with 84% being the ...
0
votes
1answer
19 views

Relation between size of parameters and complexity of model with overfitting

I'm reading the book Pattern Recognition and Machine Learning by Bishop, specifically the intro where he covers polynomial regression model. In short, let's say we generate $10$ data points using the ...
0
votes
0answers
12 views

Cox Proportional Hazard calibration with same it's meant to predict

I want to model time to churn with a Cox Proportional Hazard model and use it to predict people likely to churn. The question I have is how valid is to calibrate the model with the same people I want ...
0
votes
0answers
18 views

Will applying SVM in high dimensions (7) with limited training examples (41) likely lead to overfitting?

Right now I have a dataset with 41 training samples (and no testing samples either unfortunately). There are 7 features, but I've been treating the problem as a 2-D problem thus far (in other words, ...
0
votes
0answers
78 views

Validation loss much lower than training loss. Is my model overfitting or underfitting?

I am using a convolutional neural network to train a large set of spectral data, which has 653 classes. I am storing the loss values and the accuracy values per epoch in a list. the optimiser i am ...
1
vote
0answers
34 views

How does the number of epochs affecting GANs training?

In CNN training, increasing the number epochs would lead to overfitting. However, to train a GAN, would a too large number of training epochs matter? Indeed, I also do not understand what does it mean ...
0
votes
2answers
85 views

Why don't we use regularization on decision tree split?

I heard people ask which one is better: Linear regression with regularization or Random Forest. My question is why can't you use regularization with Random Forest? My understanding is that different ...
0
votes
1answer
42 views

Preprocessing: Why do we remove constant/invariant predictors?

For a recent research project we used machine-learning. In the preprocessing phase we removed 2 predictors because they contained mostly uninformative 0 values (x1 = 100%, x2 = 99%). Is it the right ...
0
votes
0answers
24 views

Does an aberration imply overfitting?

I was reading-up on over-fitting for my project with a small dataset and it's clear that fluctuations in validation loss and accuracy imply overfitting, but does that include constant oscillation or ...
1
vote
2answers
38 views

Is it possible to test overfitting with randomized data?

I have built machine learning models for a classification problem with four classes. They run at around 70% nested cross validation accuracy. I am looking to do further testing to check of ...
0
votes
0answers
29 views

Way to stop model from overfitting in automated training pipeline?

I'm currently training a gradient boosting model for which I want to create an automated training pipeline containing hyperparameter optimization with hyperopt and also cross-validation. While trying ...
1
vote
1answer
67 views

Number of Covariates in Cox PH Model and Overfitting

I have a small time to event dataset (N=20) where patients are given one of two drugs (drug) at varying doses (...
1
vote
1answer
27 views

Rule of thumb Overfit in a MLP or is it possible with N = 135

I want to say ahead that I highly appreciate any Literature recommendation (Book, blogg, ...) besides ESL and ISL. My Question: Is it possible to train a 3 layer multilayer perceptron (mlp) in a ...
0
votes
2answers
24 views

Standard Deviation (SD) as additional metric for model evaluation?

I wanted to ask whether you think that it can be useful to compare the standard deviation of the predictions (not the standard error!) in addition to other metrics like RMSE to get an idea on the ...
4
votes
2answers
1k views

Can K-fold cross validation cause overfitting?

I am learning $k$-fold cross validation. Since each fold will be used to train the model (in $k$ iterations), won't that cause overfitting?
4
votes
1answer
55 views

Do out-of-sample fitting methods solve the problem of over-fitting?

Suppose we have a regression model, and we want to fit this to training data, and then make predictions on test data. There is a well-known danger that out-of-sample predictions will be poor, due to "...
18
votes
1answer
1k views

Mathematical/Algorithmic definition for overfitting

Is there a mathematical or algorithmic definition of overfitting? Often provided definitions are the classic 2-D plot of points with a line going through every single point and the validation loss ...
0
votes
0answers
37 views

What is the best evaluation of training the sequential modeling?

It's concerned with the probabilistic modeling of the sequential dataset. As far as my understanding, well-known RNN methodologies consist of two steps: firstly, train the model representing $p(y_{i}|...
0
votes
0answers
12 views

Estimating Gaussians with (Gaussians+cross entropy) or (output+MSE)?

model used :- neural nets x is an N-dimensional vector. y is a real number. p(x,y) is the joint probability distribution over x and y. I know p(y|x) is a Gaussian probability distribution for ...
0
votes
0answers
43 views

Is it possible to have overfitting due to high bias?

Overfitting is usually associated with high variance, whereas underfitting is associated with high bias. But one of my professors at uni mentioned that overfitting might be caused by high variance and/...
1
vote
0answers
17 views
1
vote
3answers
37 views

What happen if I choose the hyperparameters of a classifier based on lowest generalization error?

In this question, the OP asked about a situation that he/she combined training and test datasets into an agumented dataset and then tuned the hyperparameters for best accuracy and then use the ...
0
votes
0answers
26 views

Is the model over-fitting the data?

On the y-axis you've got RMSE and on the x-axis you've got the number of epochs. Then in blue, the validation error, in red the training error. What do you think is the optimal number of epochs ...
1
vote
0answers
48 views

Why is my keras resnet50 model overfitting? [duplicate]

I have applied Keras ResNet-50 on a small x-ray image dataset. I tried making layers both trainable and non-trainable, but my model validation accuracy doesn't improve above 50%. I don't understand ...
0
votes
0answers
34 views

Is there life after over-fitting?

At this point in a talk by Nando de Freitas, there is an answer to an audience question, about how theory has got left behind in statistics, but theory is still important, and he gives an example ...