Questions tagged [overfitting]
Modeling error (especially sampling error) instead of replicable and informative relationships among variables improves model fit statistics, but reduces parsimony, and worsens explanatory and predictive validity.
972
questions
0
votes
0
answers
15
views
In Sklearn, based on scores, what model should I start tuning? [closed]
I am quite new in the Data Science world, so please be patient.
I am working on this Titanic classification project just to start practicing with machine learning https://www.kaggle.com/competitions/...
- 1
0
votes
1
answer
49
views
Can this be considered overfitting?
I have been trying to use the LSTM model for a monthly time series with a length of 404 (384 for training and 20 for test). I created 4 pairs of training/validation sets, trained different models, and ...
1
vote
0
answers
80
views
Is perfect isotonic probability calibration realistic?
I work with a labelled tabular dataset of about 1 million observations, with the target being binary. The dataset is heavily imbalanced - about 0.5% positive class.
I have trained a gradient boosting ...
- 11
6
votes
6
answers
746
views
Confused about the notion of overfitting and noisy target function
So I am reading a textbook called "Learning from Data" by Abu Mostafa et al.
I am confused about the following concepts:
According to the authors, most real-world target functions $f$ are ...
- 1,373
0
votes
0
answers
10
views
What is the relationship between bias-variance and sensitivity-specificity for novelty detection?
An over or under-parameterized binary classification model (- vs +) tends to over or under-fit (bias-variance tradeoff). This leads to errors during prediction on unseen data. Depending on if ...
- 53
0
votes
0
answers
10
views
Getting a Many to One LSTM/MLP to overfit
I have a dataset of 20 thousand horses. For each horse, I have its 10 last historical races (finishing time, position, track name, distance etc. for 41 features) and am trying to predict its finishing ...
- 247
0
votes
1
answer
15
views
Should I be concerned of Over-Optimism in Nested Cross Validation with Multiple Scoring Criteria?
Suppose I am using Nested CV to estimate the generalized error of a model and a set of hyperparameters. I have chosen two scoring metrics of interest: Brier Score and AUC, because I want to get an ...
- 23
2
votes
1
answer
167
views
Training error loss vs "classification loss"
From the paper To understand deep learning we need to understand kernel learning, three questions:
In section 2 "Setup" there appears a definition of interpolated classifier as an algo that ...
user76943
2
votes
0
answers
24
views
Continue train xgboost specifically on misclassified observations?
I'm considering integrating the Boosting technique into a basic XGBoost classification model, in which I'd focus on misclassified instances. Assuming I have already used ...
- 121
1
vote
0
answers
20
views
Learning curve - Why does the train learning curve is flat? [duplicate]
I implemented a model in which I use Random Forest as classifier and I wanted to plot the learning curves for both training and test sets to decide what to do next in order to improve my model.
...
- 23
1
vote
1
answer
40
views
Worst choice to mitigate overfitting
Here's a multiple choice question that was asked at an exam.
You are tuning a linear classifier that you suspect is overfitting the data. Which of the following choice is most likely to aggravate the ...
0
votes
0
answers
12
views
Effect of batch normalization after input layer on model performance
I built a ML model with high accuracy using
...
- 11
1
vote
1
answer
58
views
How to Choose Polynomial Degree for Regression Model when Error Keeps Reducing As Degree Increase
I have a relatively small dataset with less than 100 samples, with one predictor and one outcome variable, both numerical. I generated models using lm and glm functions.
For linear and polynomial (2 ...
- 11
0
votes
0
answers
15
views
XGboost validation immediately drops and becomes stationary
I'm attempting to fit an xgboost model to some data. During training I'm seeing the RMSE for the validation set very quickly decrease, and then become basically static. The Validation performance is ...
- 1
1
vote
0
answers
46
views
Model calibration in overfitted models
Why in Shrinkage, due to an overfitted prediction model, do we tend to overestimate risk for "high risk" subjects and to underestimate risk for "low risk" subjects ?
Intuitively I ...
- 25
1
vote
0
answers
10
views
Assessing the Penalties of Overparametrization when employing a k-fold cross-validation
I am currently conducting a study using spatial occupancy models. To select the most predictive model, I am employing a k-fold cross-validation approach. I am aware of the potential pitfalls of over-...
- 11
0
votes
0
answers
46
views
Model returns low training F1 Score, but high Testing and Validation F1 score
I am currently working on a very imbalanced dataset:
24 million transactions (rows of data)
30,000 fraudulent transactions
(0.1% of total transactions)
and I am using XGBoost as the model to predict ...
- 11
3
votes
1
answer
55
views
Random Forest with Test Accuracy of 1
I am running a random forest for classification of a data with three classes in R, and each class has around 20 samples. I am partitioning data into train and test in 80:20 ratio using caret package. ...
- 31
0
votes
1
answer
61
views
Overfitting in Cox PH model
I'm conducting a survival analysis using Cox proportional hazards model. The failure in the analysis is crime.
I have a binary covariate (1 = yes, 0 = no) for which I get huge hazard ratio – usually ...
- 25
0
votes
1
answer
70
views
Is my model overfitting?
I am currently working on a very imbalanced dataset:
24 million transactions (rows of data)
30,000 fraudulent transactions (0.1% of total transactions)
The dataset is split via Year, into three sets ...
- 11
3
votes
2
answers
119
views
When does model selection begin to overfit?
Suppose you have a small dataset (perhaps 1000 labels), and you are using cross-validation to train different models and to choose the best one (according to their cross-validation scores).
It seems ...
- 1,273
1
vote
0
answers
8
views
What are the relations between overfitting and susceptibility to adversarial perturbations in classification?
For simplicity I am asking this question for a binary classification problem. In the non-robust case, for a dataset $(x_i, f(x_i))$, $i=1,\dots N$ with $x_i$ drawn iid from a distribution $D$, we ...
- 11
0
votes
0
answers
38
views
Class-Imbalance: How to handle different class distributions in training and held-out test data?
My dataset is high dimensional (sample size is 200 with 300 features) and imbalanced. The imbalance ratio is 80:20 in the training set and 88:12 in the held-out test set (collected at a different time ...
- 145
0
votes
0
answers
19
views
Huge overfit on prediction model -due to data with low predictive power or can this be fixed? (Python)
I am not used to working with machine learning models, and are currently sitting with an issue i hope you can help me with.
I am sitting with a multi classification problem, where i try to predict ...
- 1
0
votes
0
answers
23
views
Why there is no need for a test dataset when using bayesian inference methods? [duplicate]
In a comment one user said that the true guard against overfitting is the adopted priors but, for example, in bayesian neural networks we still have priors on the weights and the common advice is to ...
- 325
1
vote
1
answer
51
views
Questions on what it means when we talk about "overfit"
"Overfit" is a commonly discussed concept in ML community. However, I tend to feel that there might be abuse of using this terminology. I wonder what it means when we talk about overfit, ...
- 61
0
votes
0
answers
31
views
100% training and test accuracy in binary classification task
I'm currently reproducing a method on my data set. In the literature, training and test accuracies are generally high, mostly between 90% to 99%. I get a training accuracy of 100% and a test accuracy ...
- 81
1
vote
0
answers
28
views
Why are some models performing better with a larger data set, and why are some models performing worse?
Background
I am doing a project where I am comparing 5 models. It is a regression problem, where the model uses the 3D structure data of a protein and a drug molecule bound to it (specifically, the ...
0
votes
1
answer
82
views
Why is my polynomial regression with gradient descent not overfitting?
I wanted to implement linear regression with gradient descent from scratch and demonstrate how you can overfit when using too many polynomials. Unfortunately my model does not really overfit the data. ...
- 107
1
vote
1
answer
95
views
Sign of Overfitting from a Confusion Matrix
I have used RandomForestClassifier from Sklearn to solve a multiclass classification problem (12 classes in total). I get my x ...
- 137
0
votes
0
answers
12
views
About the performance of a model after changing the class attribute
I want to build a model using a dataset. Then, I edit the dataset by changing the class attribute (let's say I will have a new version of the dataset). After that, I want to apply the same model to ...
- 11
0
votes
0
answers
101
views
LightGBM accuracy not increasing with iterations on Validation Set?
I am training a model with LightGBM, and I am getting an output like this:
...
- 235
2
votes
2
answers
57
views
Does the best model necessarily have the best results on validation set?
During the fine-tunning of a DistilBert model, I tried two optimizers (with different parameter sets) on the same dataset.
Here are the results:
...
- 23
1
vote
1
answer
509
views
Is there a risk of overfitting when hyperparameter tuning a model
Is there a risk of overfitting when hyperparameter tuning a model using Optuna (or another hyperparameter tuning method ), with evaluation on a validation set and a large number of trials?
While a ...
- 105
0
votes
1
answer
88
views
Is repeated hyperparameter tuning can lead to overfitting?
I'm performing hyperparameter tuning for a classifier. After I finish, I'm updating the hyperparameter search space and re-tuning the hyperparameters again. I repeat this process a few times. In ...
- 105
0
votes
0
answers
39
views
Why overfitting is a problem for neural networks but not for models fitted through MCMC methods?
why for neural networks it is advised to set a test dataset apart to check if it overfitted while for statistical models fitted through mcmc this is never done?
if a model has too many parameters ...
- 325
0
votes
0
answers
17
views
Variable importance calculation in perfect fitted model (Y = x1+x2+x3) - which method?
Assume I have a dependent variable (Y) that is the sum of 5 independent variables (x1, x2, ..., x5) and with no error.
I can subset my data into three groups with a grouping independent variable (g).
...
- 1
3
votes
1
answer
120
views
Second differences notion behind GAM penalties
I'm going back through Simon Wood's book on generalized additive models (GAMs) and came back across the definition of the penalty term employed, which is supposed to combat overfitting of smooths. The ...
- 6,021
1
vote
1
answer
84
views
0
votes
0
answers
34
views
Loss indicates Underfitting, Metrics indicate overfitting. What now?
While fine-tuning a deep neural network I ran into the following situation:
My train- and validation loss are both decreasing and have very similar values throughout training. Especially the train-...
- 1
0
votes
0
answers
49
views
When does target encoding lead to overfitting
Let us say, we are tasked with setting (average/list) prices that are likely to convert for heterogeneous products (e.g. used cars of all shape and sizes - made up example!). Let us also say that we ...
- 1,957
0
votes
0
answers
19
views
Why does making new features increase the validation accuracy but decrease the test accuracy?
I am working on a competition on kaggle. The competition is a classification problem. I tried to extract 2 new features(engineer features) from the data.
The accuracy on the validation data increased ...
- 1,360
0
votes
0
answers
18
views
Rule of thumb (10obs:1fixed effect) to avoid overfitting GLMMs
This question has been asked elsewhere (e.g., here) but has not yet been adequately answered.
I have a nested dataset of annual bird counts (n=5 response variables) across 12 years taken from 31 ...
- 33
1
vote
0
answers
59
views
In-sample $R^2$ vs true asymptotic $R^2$ for multiple linear regression
In this paper, Yarkoni & Westfall (2017) present the next example to illustrate over-fitting and the differences between goodness of fit and test error:
"When are the problems of over=fitting ...
- 31
1
vote
1
answer
71
views
What do I make of all classification scores being equal to 1?
I've built an XGBoost classifier with following code:
...
- 111
1
vote
0
answers
50
views
Is my model overparameterized? Considering regularization but unsure how to keep some significance score (or if I even should)
I have trained a model on some sales data, we have separated the data into different groups of customers based on the marketing strategies of our client. We have a total of 10 groups and 31 coupons. ...
- 368
2
votes
1
answer
71
views
Bootstrap in cluster experiments
I am planning an AB-test for something like a call-center. We are testing a new interface for the call-center operators. It has to be tested within a small group of roughly 20 operators. The main ...
- 21
2
votes
1
answer
577
views
Overfitting using lightGBM?
I have a small dataset composed of 800 data points where I need to perform a regression task. I randomly chose 10% of the dataset to be used as validation.
The problem is that I am not sure if I am ...
- 371
1
vote
1
answer
142
views
What does the number of support vectors tell us?
I've trained a SVM regression model, and noticed a large number of support vectors required. I have found the following discussions where it seems people are concluding that a larger number of support ...
- 53
4
votes
3
answers
656
views
Why would a smaller AIC than BIC lead to an increased chance of both overfitting and underfitting?
I am puzzled by the following statement in my lecture notes
AIC penalty smaller than BIC; increased chance of overfitting
BIC penalty bigger than AIC; increased chance of underfitting
Is there a ...
- 641