Questions tagged [overfitting]

Modeling error (especially sampling error) instead of replicable and informative relationships among variables improves model fit statistics, but reduces parsimony, and worsens explanatory and predictive validity.

Filter by
Sorted by
Tagged with
2
votes
1answer
58 views

Is there a better way to describe a model's generalization performance than “under” and “overfitting”?

To me, under and overfitting are the two of the most vague concepts in machine learning. From Google's first link when you look up these definitions. A model is said to be underfitted if it "...
0
votes
0answers
18 views

What is the essence of Linear Discriminant Analysis while considering the correlated features for Inter Class problem?

Suppose, I have samples from APPLE, MANGO, and ORANGE --- these 3 classes. The goal is to do multiclass classification. Now, let's say, I have calculated 4 features from all of the 3 classes. By rules ...
0
votes
0answers
9 views

Does using a grouper algorithm in an explanatory or causal inference model use up degrees of freedom?

For example, say I'm performing a regression to explain how much various factors (age, sex, diagnosis, procedure) affect total annual medical cost among patients. There are thousands of diagnosis and ...
0
votes
0answers
5 views

How to properly retrain a model with quantization aware training

I am trying to tune a model via quantization aware training (QAT). The model is from rcmalli. It is a ResNet50 architecture. The model was trained by them on the vggface2 dataset. I use the model to ...
2
votes
1answer
30 views

Overfitting small dataset necessary for deep NNs when training with big dataset works?

In the CS231n course from Standford, they state that a network should be able to overfit a small dataset by getting zero cost, otherwise it is not worth training. However, what if a network is not ...
0
votes
0answers
8 views

Train-test score similar in regression but very different in random forests

I have a multiclass classification problem, for which I wanted to try different methods. I tried multinomial logistic regression, random forests and XGBoost. I evaluated the methods with the same ...
3
votes
2answers
36 views

Should we use AUC as an indicator of overfitting when dataset is highly imbalanced?

In my problem, there are 2 class labels, but one label only counts for 1% of the total data. I first divided my data set by train_test_split such that only 10% are test set, then I performed 10-Fold ...
2
votes
2answers
58 views

Cross validation on a single model (not model comparison)

I understand the method of cross validation to be to leave out some part of a dataset (whether that be one data point at a time = LOO, or subsets = K fold), and train the model on some data, test the ...
0
votes
0answers
11 views

How to know when a random forest model is extrapolating in multidimensional space, while ignoring uninformative variables?

Lets say I train a random forest model to predict dependent variable y as a function of x1, ...
1
vote
0answers
13 views

How can one use Grid Search without overfitting the model?

I checked several questions, like Overfitting during model selection - AutoML vs Grid search and Hyperparameter tuning using grid search/randomised search, but I don't think any of them answer my ...
1
vote
1answer
19 views

What to do after knowing the model is overfitted?

So I was trying to run a model using scikit-learn. In order to tune the hyperparameters, I used RandomizedSearchCV, just like this: ...
0
votes
0answers
12 views

Am I interpreting correctly this NMF analysis?

I have to analyse a set of biological data and I am applying a Non-Negative Matrix Factorization (NMF) Approach. Given a 366 x 144 dataset, I am reasoning about overfitting and the correct rank r to ...
0
votes
0answers
22 views

Getting bad results when testing my prediction model on new patients

I analyze medical diagnostic labeled data: the independent variables are medical parameters (assume that one of the variables is blood pressure) and the dependent variable is 1 or 0. The data contains ...
2
votes
2answers
69 views

Neural Network vs regression in prediction

I collected a sample of 600 observation (time series data) with 100 predictors variables in order to predict another one. I want to use some prediction models but I know that, unfortunately, ...
0
votes
0answers
15 views

Why an increasing validation loss and validation accuracy signifies overfitting?

When I train a neural network, I observe an increasing validation loss, while at the same time, the validation accuracy is also increased. I have read explanations related to the phenomenon, and it ...
0
votes
0answers
33 views

What is the limit to consider something is overfitting?

I'm running random forests for imbalanced multiclass classification. Because of this, I'm trying many variations of RF: basic RF, balanced RF, weighted RF, undersampled RF and SMOTE RF (oversampled). ...
24
votes
2answers
2k views

Can overfitting and underfitting occur simultaneously?

I am trying to understand overfitting and underfitting better. Consider a data generating process (DGP) $$ Y=f(X)+\varepsilon $$ where $f(\cdot)$ is a deterministic function, $X$ are some regressors ...
1
vote
0answers
15 views

Is cross-validation (test) error below chance an indicator of overfitting?

I am training a binary classifier (e.g. logistic regression) on some multidimensional problem. I have tried leave-one-out and k-fold cross-validation. I have tried L1 and L2 regularization, and I have ...
1
vote
1answer
27 views

What is the best way to overfit a model with accuracy?

I need to overfit a model to specific data.(These data have been created from simulation of complex multidimensional models.) The final goal is to extract coefficients (betas) to retrieve with ...
1
vote
0answers
33 views

Random Forest with train AUC = 1 and test AUC = 58%

I'm trying to understand why my train AUC = 1 while my test AUC is near 58% using random forest. Context: You are trying to sell a product, and you have historic data about the purchases/noPurchases ...
1
vote
1answer
46 views

Overfit in aggregated models: boosting versus simple bagging

Let's fix a bagging setup, where several models are build independently and than somehow aggregated. It is intuitive that increasing the number of weak learners ( N ) does not lead to overfit ( in the ...
6
votes
3answers
136 views

Impossible to overfit when the data generating process is deterministic?

For a stochastic data generating process (DGP) $$ Y=f(X)+\varepsilon $$ and a model producing a point prediction $$ \hat{Y}=\hat{f}(X), $$ the bias-variance decomposition is \begin{align} \text{Err}(...
0
votes
0answers
36 views

Can I change the rows / composition of a dataset without violating modeling assumptions?

I am building a few models to explain an outcome (let's say customer making a purchase during a visit to a site) from a dataset of ~200k rows with ~66 features (after dummying out categorical ...
0
votes
1answer
23 views

Why is my KNN model overfitting?

I have 20K voters geocoded and labelled with 4 classes of voting intention. I want to predict based on neighbors intention which looks correlated on a map. Using K nearest neighbors with lat/lon ...
0
votes
0answers
16 views

Is it an overfitting problem for SVM classification?

I am new in Machine learning, and I want to detect emotions from the face. Preprocessing: I used equalizeHist to equalizes the histogram of grayscale images (JAFFE database with 213 images), in the ...
1
vote
0answers
28 views

Doubly Robust Estimator

When use Doubly Robust Estimator we train m0/m1 models and propensity score model to be used by the estimator. Is it OK to use the same dataset to train those models and then use them to measure ATE ...
1
vote
1answer
23 views

Can I still use an overfitted model with high test accuracy?

Below is the training statistics output from training a Keras/TF model. You can see val_accuracy peaks at Epoch 4 with 0.6633. After that accuracy(train) continues to go up but val_accuracy becomes ...
1
vote
0answers
42 views

Is it possible that accuracy testing and the accuracy training are 1.00 in binary classification [closed]

I am working to improve classification results with more ML algorithm. I get 100 percent accuracy in both test and training set. I used GradientBoostingClassifier, XGboost , RandomForest and Xgboost ...
0
votes
1answer
23 views

How to split UNSW-NB15 dataset (full dataset) to training/validating/testing set for training neural network properly?

I'm working on classifying UNSW-NB15 dataset into 2 categories (benign - malware) using neural network. The full dataset include about 2.000.000 benign samples and 300000 malware samples. I'm assuming ...
0
votes
1answer
30 views

Data leakage with clustered observations

I have (what I call) a clustered dataset, that is: for one client, I can have multiple observations that will have some variables in common and some variables will be specific to each observation. ...
2
votes
1answer
40 views

Regression: is it wrong to bin a continuous variable to overcome overfitting?

Would statisticians hang me for doing the following? I have a heterogeneous dataset of elderly subjects. Thus, I have model with 7 predictors, including 4 categorical ones, of which some have many ...
0
votes
0answers
31 views

dealing with over-fitting in audio classification

I am following this tutorial for audio classification. My results are correlated with the author's results and I am getting $99\%$ accuracy for the train set and $91.8\%$ accuracy for the test. I have ...
0
votes
0answers
18 views

Should I worry about in-fit overfitting if out-of-fit accuracy is maximized?

I typically find that random forest always overfits to some degree on the training data. That is, the in-fit R2 is typically substantially higher than the out-of-fit, cross-validated R2. In general, ...
0
votes
0answers
16 views

best hypothesis for data with zero mean noise is the one which assumes no noise at all?

In an ml-class, they introduced overfitting with an example: Say, we have $x$ picked from $Uniform(0,1)$ $v$ random noise, picked indepidently of $x$ from $Uniform(-0.3,0.3)$ and $\mathbb{E}[v] = 0$ $...
4
votes
1answer
42 views

Does Dropout need a validation set to prevent overfitting? [closed]

Is it really necessary to use a validation set to avoide overfitting while we are using Dropout ?
3
votes
0answers
32 views

How can validation loss increases than slowly decreases when overfitting?

I'm using CNN to do multi-regression jobs, but some wired results confuse me. Here's my training results curves: As you can see, the training loss first decreases fastly and then converges, while ...
1
vote
1answer
22 views

Does my validation curve indicate over fitting from train and test?

I carried out a grid search on my xgboost and varied the parameters below. I noticed in my grid search results that the train score is very high, e.g. 0.99999 and my test scores are more modest around ...
0
votes
1answer
24 views

Data Perturbation - Model robustness test

I came across with presentation about robustness test recently and I didn't exactly understand how to apply it to ML model (not DL). The presentation show a graph: x-axis - some metric, for example ...
0
votes
0answers
17 views

Segmentation not able to overfit with small batch of images - Dice score max 0.7

Based on most online resources for debugging deep learning models, the first recommended step is overfitting a simple model on a very small batch of images. I am training a UNet model on a very small ...
0
votes
0answers
35 views

Python Random Forest Prediction Probabilities Reliability, Overfitting?

TLDR: RF prediction probabilities are not consistent I have created a calibrated Random Forest Model to predict probabilities for attrition of the workforce, but what I am finding is that ...
0
votes
0answers
14 views

ConvNet - What to improve regarding architecture, procedure and technique?

I have a dataset of 180k images of license plates (so, not necessary to localize the license plate at first) for which I try to recognize the characters on the images (License plate recognition). All ...
0
votes
3answers
102 views

Is this overfitting?

The Use Case: We are given three unique, 'ground truth' binary training patterns (not patterns with 'noise'). A machine is to be trained with these three vectors. The requirement is that once trained, ...
4
votes
1answer
83 views

Does using Cross-Validation give you the green light to do exhaustive hyper-parameter searches?

By hyper-parameters I mean not only the machine learning algorithm hyper-parameters (learning rate, etc.), but also hyper-parameters like "what's the ideal number of data points to use" or &...
2
votes
1answer
51 views

What could be the reasons that making validation loss jumping up and down?

I am building some image classification model with reasonable size data (~3K) images in both training and validation set. However, I noticed the performance on validation set is not stable. For ...
0
votes
0answers
31 views

Why is SSD object detection model overfitting even with very high weight decay?

Actually I am struggling for a long time with this problem and had tried a lot of experiments. I am working on an object detection model using the SSD architecture with various backbones. I started ...
0
votes
0answers
21 views

Linear regression overfitting and regulization

When creating a linear model with many variables, there can be overfitting. Let's say that the training error is 10, and the testing error is 12. So one can use Ridge or Lasso regression to used the ...
1
vote
0answers
12 views

How do you cope with the risk of false-positives in exploratory analysis?

Let's say that I'm running exploratory analysis on a dataset. For instance, let's say that the dataset consists of several features and two groups and I want to see which features are significantly ...
1
vote
0answers
16 views

If overfitting occurs, what should i focus more? [duplicate]

I'm a graduate student who are studying AI. I have constructed one model for voice classification but, overfitting occurred. I tried a lot to overcome this phenomenon. (i.e. weight standardization, ...
0
votes
1answer
41 views

The accuracy decreased when I used L2 regularization method [closed]

I am training a 6 layer Deep Neural Network with: ...
7
votes
4answers
248 views

Relationship between overfitting and robustness to outliers

What's the relationship between overfitting and sensitivity to outliers? For example: Does robustness to outliers make necessarily models less prone to overfitting? What about the other way around? ...

1
2 3 4 5
15