Questions tagged [overfitting]

Modeling error (especially sampling error) instead of replicable and informative relationships among variables improves model fit statistics, but reduces parsimony, and worsens explanatory and predictive validity.

Filter by
Sorted by
Tagged with
1 vote
1 answer
15 views

Do I need to normalize data before applying L1, L2 norm in ANN

I wish to train the ANN and use regularizers to avoid overfitting. I need some suggestions, is it mandatory to normalize the data before using L1, L2 regularizers. I would highly appreciate if you can ...
user avatar
  • 141
3 votes
1 answer
37 views

Matrix Factorization and Overfitting

I recently came accross the algorithm of Matrix Factorization for a recommendations system. One of the tutorials I followed can be found here. According to it given the initial matrix $R$ and the ...
user avatar
0 votes
0 answers
6 views

Learning Average Dependent 0/1 Variable

Suppose I have a matrix $X$ and a dependent vector $y$ whose entries are each in $\{0, 1\}$ dependent on the corresponding row of $X$ Given this dataset,I'd like to learn a model, so that given some ...
user avatar
  • 4,440
0 votes
1 answer
63 views

Low classification accuracy

I want to do a multi class classification with 6 classes. Whole dataset has 12750 and 56 features samples, so every class has 2125 samples. Before prediction I reduces amount of outliers by ...
user avatar
  • 31
3 votes
1 answer
70 views

Should stepwise regressions or overfitting also be avoided for exploratory (hypothesis generating) modelling?

In a recent paper, Andrew Tredennick and colleagues (2021) suggested to use the drop1() function in R for exploratory modelling (that is to generate new hypotheses ...
user avatar
  • 383
0 votes
0 answers
12 views

Effect of duplicate/redundant labels on performance of model

I am training a CNN to predict age,mass and tone from images. The structure of my dateset is as follows ...
user avatar
0 votes
0 answers
36 views

How variable alpha changes SGDRegressor behavior for outlier?

I am using SGDRegressor with a constant learning rate and default loss function. I am curious to know how changing the alpha parameter in the function from 0.0001 to 100 will change regressor behavior....
user avatar
13 votes
3 answers
2k views

If I use a regularization (e.g. L2) can I not apply early stopping?

I've seen that early stopping is a form of regularization that limits the movement of the parameters $\theta$ in a similar way that L2 Regularization penalizes the movement of $\theta$ to be closer to ...
user avatar
  • 287
0 votes
2 answers
114 views

Is it possible to evaluate a given model without having access to its fit method?

I have a data set with one real-valued feature and a real-valued target. Someone has used this data set to fit a model (a regression). I get a results of this fit, which is a single function mapping ...
user avatar
  • 463
0 votes
0 answers
14 views

Regressor-based L2 penalty [duplicate]

I'm working on a multiple regression problem where I have reasons to believe some (if not all) of the regressors have been cherry picked/data mined to a varying degree. My hypotheses are that there's ...
user avatar
  • 749
8 votes
2 answers
256 views

PCA as a Cure for the Curse of Dimensionality

I would like some clarification as to how principal component analysis mitigates the Curse of Dimensionality problem. My particular interest is in curbing overfitting in my modelling, or more ...
user avatar
2 votes
2 answers
83 views

Why use regularization instead of feature selection for logistic regression? [duplicate]

For a non-linearly separable problem, when there are enough features, we can make the data linearly separable. It seems to me that for logistic regression, the reason of overfitting is always ...
user avatar
2 votes
1 answer
114 views

Does higher variance in predictions result in higher variance error estimation?

Problem Assume: $\forall i \neq j, Y_i \perp Y_j$. This is just a usual independence assumption. $\forall i, j, Y_i \perp \hat{Y}_j$. This should come from the fact that outcomes (in our evaluation ...
user avatar
1 vote
1 answer
42 views

Almost duplicate samples between train/test: overfitting?

I have been thinking about this for a few so I would like to hear some opinions. It could be complicated to explain so I will update the question if there is something that its not clear. Imagine I ...
user avatar
1 vote
0 answers
27 views

Deep learning model comparison

I have a question regarding my deep learning model, I have trained 2 models with different hyperparameters, the second model got loss results higher than the first model, first model loss = 0.0047057-...
user avatar
  • 11
4 votes
1 answer
125 views

Why custom evaluation/scoring metric is causing overfitting in (cross) validation?

I am using machine learning to approach a balanced binary classification task. Some of rows are more important/valuable than others, so getting them right is extra important. Therefore, to accommodate ...
user avatar
1 vote
0 answers
26 views

Why does one model overfits faster than another?

Lets' say we have 2 models and we train both of them on the same data for the same number of epochs. When we monitor the validation loss, we realize that the point where the validation loss stops ...
user avatar
  • 143
1 vote
0 answers
44 views

Why can't OLS estimates be used to obtain regression parameters when dealing with high dimensional data?

Suppose I have a data set consisting of $n$ observations: ${\displaystyle \left\{\mathbf {x} _{i},y_{i}\right\}_{i=1}^{n}}$. If I apply linear regression :${\displaystyle \mathbf {y} =\mathrm {X} {\...
user avatar
1 vote
1 answer
42 views

How to study the overfitting of a classification model in my research paper?

I do research on a dataset having texts, I split them into 80% training and 20% for testing. I submitted my research article to a journal and the reviewers responded with major revisions that include ...
user avatar
  • 13
0 votes
2 answers
156 views

My training accuracy is 1.0 and my test accuracy is 0.994. Am I overfitting for a multiclass classification?

This is a multiclass classification for an imbalanced dataset. I set the class_weight for this model to "balanced". I have a perfect training accuracy (1.0) and a nearly perfect testing ...
user avatar
1 vote
2 answers
82 views

How to compare the random forest performance using R-Squared?

I've trained a random forests for a regression problem. Now, I want to check if the model is not overfitted. I have tuned the parameters and then compared the R-Squared of Train and Test dataset as ...
user avatar
2 votes
0 answers
59 views

Log transformed response variable and overfitting issues in random forest regression

I am trying to fit random forest regression for continuous skewed response variable with so many zeros. Here is the plot of the response variable: In An Introduction to Statistical Learning, authors ...
user avatar
0 votes
0 answers
82 views

How to interpret RMSE of 0 with a lot of features

I have 122 features for a regression problem. Here are some stats on a random forest model using RMSE: With no scaling or dimensionality reduction: Train RMSE: 0.0 Test RMSE: 0.0 CV RMSE: 0.0 With ...
user avatar
0 votes
0 answers
6 views

Why training accuracy is better? [duplicate]

I'm still new in this field. I made a classification with random forest and a balance dataset with pos and neg label. The training score is 100% but the testing score is 80%. Why is there such a big ...
user avatar
1 vote
1 answer
26 views

Determining overfitting model by computing variance in prediction error

I have a data set for regression, with a set of input features and 1 response variable. To confirm if a trained model has overfitted, we can see if the train error << test error at untrained ...
user avatar
  • 167
0 votes
0 answers
23 views

Is there an explanation for a classifier achieving high F1 scores, but having still high CrossEntropyLoss?

I am training a CNN classifier on a balanced dataset (around 35k examples for each label) with 13 classes. The model seems to achieve high F1 scores from the first batches; The F1 score for each class ...
user avatar
  • 3
0 votes
0 answers
19 views

Can I use RMSE to diagnose overfitting in a Bayesian Calibration?

I am fitting a simulation model using Bayesian Calibration (DREAMZS MCMC using the BayesianTools packages in R). I have several time series I am calibrating to (e.g. log stream flow and nitrate ...
user avatar
0 votes
0 answers
14 views

Verifying if an already tuned model is overfitting or underfitting

Image that you are given a regression model $f_\theta(x)$ where the model parameters and have already been tuned and trained. How do you assess if this model is overfitting or underfitting ? You are ...
user avatar
7 votes
4 answers
911 views

Why do we say that the model has a high variance when variance is actually the measure of spread of the data and not some property of the model?

I am trying to understand the difference between bias-variance and overfitting-underfitting. If a modal overfits the data it means that it will not generalize well on new data because it over learns ...
user avatar
1 vote
0 answers
19 views

Avoiding Overfitting by making Parameters Sum to 0

I'm following a blog that is modelling soccer scores using a parameterised poisson process. Each team has an attack rating $\alpha_{i}$ and defence rating $\beta_{i}$. When coding up the model and ...
user avatar
  • 123
1 vote
0 answers
15 views

What is possibly wrong in this Validation Accuracy and Training Accuracy [duplicate]

As general perception over training and validation accuracy is that if training accuracy is high and validation accuracy is marginally low, then it is most probably over fitting. Consider a case of ...
user avatar
0 votes
0 answers
10 views

Can an Random Forest regression overfitted model underfit after random search?

I trained the basic random forest regression model in sklearn and it overfits. The dataset size is about 4500 rows with a dozed of variables. Example ...
user avatar
0 votes
0 answers
15 views

Overfitting in ordinal logistic regression in SPSS

I have a question about overfitting in ordinal logistic regression in SPSS. Can I check presence of overfitting in ordinal logistic regression in SPSS. I know that AIC, BIC were used for presence of ...
user avatar
  • 1
0 votes
0 answers
87 views

How to correctly diagnose overfitting using all information: training set, validation set and test set?

I understand that overfitting is typically defined/described as the relationship between training set error and test set error - that overfitting is when a model performs significantly worse out-of-...
user avatar
0 votes
2 answers
31 views

What degree of difference does validation and training loss need to have to be called overfit?

I've trained an LSTM network to predict time series data however i'm quite new to LSTMs and am unsure if the model has overfit. I know that an increasing validation loss relative to a decreasing ...
user avatar
1 vote
0 answers
62 views

Is it correct to train and validate the model on F1-score metrics?

I am trying to do experiments on multiple data sets. Some are more imbalanced than others. Now, in order to assure fair reporting, we compute F1-Score on test data. In most machine learning models, we ...
user avatar
  • 121
2 votes
1 answer
125 views

How to test if a linear mixed model (mixedlm) is overfitting in Python?

I've read several threads about overfitting but still couldn't find a reply that seems to be applicable to my case... I use statsmodels' mixedlm to try studying the effects of 4 predictors on ...
user avatar
0 votes
0 answers
23 views

How to assess LMM performance in a new data set?

This question directly relates to this question and generally only differs in the fact that I don't want to fit a GLMM but rather a "regular" Linear Mixed Model. The given answer on using ...
user avatar
  • 75
0 votes
0 answers
19 views

Comparison between regression models based on trees

I'm solving a prediction problem in which I have an independent variables Y and 13 dependent variables which also are highly correlated. My dataset is composed by 124 observation for the train dataset ...
user avatar
  • 1
0 votes
1 answer
49 views

Bias-variance trade-off in linear regression [closed]

As it’s understood, in the bias-variance trade-off, variance refers to overfitting of the model and it examines the variability of output predictions. Suppose we have a simple dataset with one ...
user avatar
0 votes
1 answer
175 views

How to identify overfitting from LSTM plot, from the prediction on trained+unseen data

I am currently learning LSTM-RNN models and I have done some tests to see how they work. As in the most NN, overfitting and underfitting is a problem in ML. I have read articles such as this guy here: ...
user avatar
19 votes
5 answers
3k views

Why is controlling for too many variables considered harmful?

I am trying to understand the point of the second panel in the following xkcd comic: Specifically, how can one be misled by controlling too many confounding variables in one's models? Any pointers to ...
user avatar
  • 361
1 vote
0 answers
31 views

Underfitting, overfitting or good fit?

I am working with a medical dataset, trying to figure out people with diabetes. I implemented LSTM since the target variable is a categorical one, the accuracy gave %84. I plotted the accuracy ...
user avatar
1 vote
0 answers
29 views

Training loss, validation loss and WER decrease, then increase [duplicate]

I am trying to use Hugginface Datasets for speech recognition using transformers using this tutorial, epochs=30, steps=400, train_batch_size=16. Training loss, validation loss and WER decrease, and ...
user avatar
30 votes
9 answers
4k views

How can we explain the "bad reputation" of higher-order polynomials?

We all must have heard it by now - when we start learning about statistical models overfitting data, the first example we are often given is about "polynomial functions" (e.g., see the ...
user avatar
  • 5,718
0 votes
1 answer
36 views

Does it make sense to get the best Kth-fold CV test result from an epoch where train result were bad?

I have been looking for some explanation that could convince me over the right way of thinking about CV. My challenge is related to the automation of the model configuration process due to same kind ...
user avatar
  • 3
0 votes
0 answers
50 views

Multivariate linear regression. Is it ok to include categorical variables with some levels that only have few entries?

I am trying to modell car price data using different variables horsepower, car brand etc. My concern is with adding categorical variables that have few entries in some levels. For example let's say I ...
user avatar
0 votes
2 answers
37 views

How to estimate the probability of LOOCV error of one model to be better then LOOCV error of the correct model?

Lets consider a simple regression problem in which we have only one real-valued feature and one real valued-target. We try to fit the data using a polynomial function. We also try to use the given ...
user avatar
  • 463
2 votes
1 answer
20 views

Model selection in presence of overfitting - better test or closer train

Suppose I have a tree-based model (Random Forest for the sake of the example) and I play with a regularization parameter (tree depth) to fight overfitting. Eventually I can come up with two models - ...
user avatar
  • 318
4 votes
2 answers
117 views

Does a targets-permutation test prove that regression find a real pattern?

I need to solve a standard ("vanilla") regression problem meaning that I have a 2D array of real-valued features (X) and 1D array of real-valued targets (<...
user avatar
  • 463

1
2 3 4 5
18