Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [overfitting]

Modeling error (especially sampling error) instead of replicable and informative relationships among variables improves model fit statistics, but reduces parsimony, and worsens explanatory and predictive validity.

0
votes
1answer
14 views

Scoring rules for count models on: training data vs. validation data

In order to evaluate and compare count models (e.g. Poisson regression), we can calculate scoring rules (e.g. Brier Score, Dawid-Sebastiani score, etc.) which are explained here: Error metrics for ...
0
votes
0answers
10 views

Dynamic factor analysis

I have been running a DFA on n=40 time series containing percent cover estimates for a single species of algae. The time series represent 40 locations that span the geographical range of the species, ...
0
votes
0answers
10 views

How to do variable selection for Gradient boosting models like Xgboost and LightGBM

I am building a classification model with about ~110 variables and that gave me an AUC of about 71.96 on validation. I added about 10 more features and my AUC value decreased to 71.56 (which led to ...
1
vote
2answers
78 views

What is the difference between overfitting and “not learning”

I am trying to build a Random Forests (RF) model using around 2000 observations and a number of features (can be 50 or can 1000, I still do not know which features are to be used). One way to ...
1
vote
1answer
40 views

Can we use a mixture of normal distributions while optimising likelihood?

Let's assume that we generate some values by a mixture of two Gaussians. Now we want to find the parameters of the two Gaussians by likelihood maximisation. One good expect that the optimisation will ...
0
votes
0answers
20 views

Controlling over-fitting in local cross-validation LightGBM

I am training a lightgbm model on a binary problem (~20% of events) with below parameters: ...
7
votes
2answers
1k views

How does one most easily overfit?

This is a weird question, I know. I'm just a noob and trying to learn about different classifier options and how they work. So I'm asking the question: Given a dataset of n1-dimensions and n2-...
1
vote
0answers
41 views

Cross-Validation on a multiple linear regression model, negative values?

I'm trying to demonstrate that, using a linear model with too many predictors, that the correlation can be artificially inflated, and that k-fold cross validation can expose overfitting. To do this, ...
5
votes
1answer
69 views

Can small SGD batch size lead to faster overfitting?

I have feedforward neural net, trained on cca 34k samples and tested on 8k samples. There is 139 features in dataset. The ANN does classification between two labels, 0 and 1, so I am using sigmoid ...
0
votes
1answer
23 views

Mathematical Motivation of Splitting Into Training and Testing Sets

In Learning from Data course taught by Caltech Professor Yaser Abu-Mostafa the following notation is used to describe the in sample and out of sample errors. $E_{in}[h]=\dfrac{1}{N}\sum_{n=1}^Ne(h(...
2
votes
1answer
35 views

Cross validation and over-fitting

I've read many posts on this site that claim something along the lines of "I used cross-validation to prevent over-fitting". Which leads me to my question, does cross-validation actually prevent over-...
0
votes
0answers
14 views

cross-validation: feature selection and hyperparameter tuning. Is nesting necessary?

I am a little bit confused by the use of feature selection inside a K-fold CV together with hyperparameter tuning. So I have my dataset. I split in training & test as usual, and work on training ...
0
votes
1answer
31 views

Signs of Overfitting in Precision/Recall Curve

plz look at the following figures. As you cann see the precision is always 100% no matter which threshold (x-axis in logarithmic scale) you set! Also the second figure shows that we have a perfect ...
0
votes
0answers
22 views

Training neural network with a single instance at a time

For a semantic segmentation problem attempted using neural networks, does it make sense to try achieving overfitting with single training example and then (depending on generalization error) add more ...
1
vote
0answers
14 views

como eu posso ajustar parameteros um sistema com duas equações com a função nls()? [closed]

I Ned to fit this system of equations: I'm using nls.lm(), but want to use the nls() function # rm(list= ls()) df=read.table( text =" 0 0.010000000000000 0 0....
0
votes
0answers
79 views

Very High Training Accuracy and very low Testing Accuracy CNN

I'm using 3 layer CNN with 8, 16, and 32 filters, each of size 5 X 5. I'm getting an training accuracy of 99.97%. Testing accuracy of 41.11%. Total classes: 605 Train Set: Each class has 7 samples ...
1
vote
1answer
32 views

Proof that estimator in overfitted model is still unbiased

Assume that the true population model is given by $y=x'\beta+\epsilon$, where $x$ and $\beta$ are k-dimensional vectors, and suppose that when performing a linear regression, we accidentally overfit ...
0
votes
1answer
86 views

99% on the first epoch: overfitting [closed]

I am working with time-series data and I am trying to classify the Fault happening in the system. The problem is no matter what I try so far, I get 99.79 validation accuracy on the very first epoch. ...
0
votes
0answers
15 views

May negative dataset cause CNN model over/under-fitting?

To put you into context, let me explain a bit what I'm trying to achieve. I'm using YOLOv3 (doesn't really matter now) convolutional neural network to detect traffic signs in full images. I'm training ...
3
votes
0answers
89 views

Overfitting in Random Forest Classifier?

I would like some help from you in a classification model that I am developing. In summary, the problem is: – Classification problem with binary outcome (0/1) – The classifier is a Random Forest ...
1
vote
1answer
31 views

Neural Networks - Difference between 1 and 2 layers?

I'm currently working on a regression problem, using neural networks to constrain parameters for a complex physical scenario. I am searching the hyperparameter space for the best model and have thus ...
0
votes
1answer
30 views

validation vs test vs training accuracy, which one to compare for claiming model overfitting?

I have read on the several answers here and on the internet that cross-validation helps to indicate that if the model will generalize well or not and about overfitting. But I am confused that which ...
0
votes
1answer
30 views

Bad performance on training set and increasing validation loss: Overfitting or underfitting?

From my understanding of over- and underfitting, the two behaviors are not completely mutually exclusive. Overfitting can pretty much always be achieved if the model has enough capacitance (is able to ...
2
votes
2answers
77 views

Is overfitting always a problem?

If I test various models and the best performing model also happens to be one that appears to be overfit, is this an issue? For example, if I have a model with 100% accuracy on the training data and ...
0
votes
1answer
55 views

How to detect if model is overfitting?

I know this question is asked billion times, but I could not really find an answer to my situation. So, I want to show all the logs of Keras model learning. The problem is I don't know if my model is ...
1
vote
1answer
12 views

Overfitting when train and test set features have identical distributions

In a hypothetical setup where train and test set features have identical distributions, is the correct to say that one can not overfit.
2
votes
0answers
47 views

Overfitting cross-validation scores

I've ran 1000 iterations of XGBClassifier parameters search using RandomizedSearchCV on the titanic dataset. That's just for context, but the question applies to any CV search method, any model and ...
1
vote
1answer
67 views

How to validate models beyond checking for overfitting

I have an unusual problem, which is that my model is performing too well and I am struggling to trust it. The data is a table of "snapshots" about moments in games of chess. For example, a game that ...
0
votes
0answers
31 views

How to prevent heteroskedastic models from overfitting?

I'm trying to fit neuroscience data using a Gaussian Process, but noticed that it behaves poisson-like (var = mean). Since classic GP models assume iid noise, I figured I could get a better fit by ...
1
vote
0answers
26 views

How can I avoid overfitting when training a model to predict the outcome of live events

I have a question about predicting the outcome of 'live' events - I.E. and event which has begun but not yet finished. Say your goal is to predict whether or not a particular player will win a series ...
0
votes
1answer
68 views

Training loss decreasing, Validation loss steady, where to stop?

In the following training scenario (Orange: training loss, Blue: validation loss), what epoch is the best time for stopping the training? Validation is almost steady as we continue training, but ...
20
votes
9answers
2k views

Overfitting and Underfitting

I have made some research about overfitting and underfitting, and I have understood what they exactly are, but I cannot find the reasons. What are the main reasons for overfitting and underfitting? ...
2
votes
1answer
67 views

What if cross-validation fails to prevent overfitting

I'm training a random forest model with AUC as performance metric. I've splitted my data to train set (70%) and test set (30%) and performed cross-validation on train set to tune the hyperparameters. ...
0
votes
0answers
48 views

Is this an example of overfitting?

I am trying to predict some future values using either KNN or regression model. I have about 9 independent variables that do not seem to have strong correlation to each other (Not completely sure ...
0
votes
1answer
15 views

How to prove High Sampling Variance in over-fitted functions

I've been reading recently about over-fitting and it is frequently related to High Sampling Variance and Low Bias characteristics. However, what is the metric used to state the High Sampling ...
0
votes
1answer
49 views

Random Forest feature selection over-fitting doubt

My objective is to find genes that can be used as biomarkers with low error. I am using Random Forest (RF) using R package randomForest and following the steps in below link as it is has similar ...
0
votes
0answers
22 views

Is it Overfitting if validation metric is improving but training metric is extremely high?

If Accuracy or AUC for 4 different values of a tuning parameter in a model, ntrees in randomforest from 500 , 1000 , 1500 , 2000 over Training Data is 0.7 , 0.8 , 0.9 , 1.0 over Test / Validation ...
0
votes
0answers
14 views

Wrong weights learned when training RBM

I'm training my RBM network and on epoch #4 I have such a filters representation (my weights matrix) But on the next iteration (fifth epoch) something went wrong and my filters became like this What ...
4
votes
2answers
229 views

Can Adjusted R squared be equal to 1?

I have a dataset with around 15 independent variables. I am using a multi-regression model to fit the dataset. For model selection, I am using a backward elimination procedure based on the p-values. ...
0
votes
1answer
230 views

How to distinguish overfitting and underfitting from the ROC AUC curve?

For model selection, one of the metric is (AUC Area Under Curve) which tell us how the models are performing and based on AUC value we can choose the best model. But how to distinguish whether a ...
0
votes
0answers
37 views

do not need many controls with big data?

I am looking at a paper which uses a large panel data, 1 million observations, a dozen variables. I recall that in a discussion another one has the following comments: In structural models like ...
0
votes
0answers
28 views

Can't get a Keras model to overfit [duplicate]

(Full disclosure, this is a follow-up to this question, which wasn't completely answered on StackOverflow) The input dataset is a time series of some stock price movement, but it might as well be ...
0
votes
1answer
29 views

Why can having more “model” parameters (weights) in neural networks lead to overfitting?

In http://cs231n.github.io/convolutional-networks/, it states that a "huge number of parameters would quickly lead to overfitting" in neural networks. I don't think I quite understand this. The ...
1
vote
3answers
130 views

In Bishop's textbook, is the example of overfitting exaggerated?

Here, the data $x$ are randomly generated, and $t$ are generated by running $x$ through a function $\sin(2\pi x)$, then Gaussian noise is added. Bishop's text then tries to fit those data using a ...
3
votes
1answer
36 views

Best strategy to maximize the prediction accuracy when p >> n

I am solving the following classification problem: thousands of features, but only 40 samples (i.e. p >> n) classes are balanced it is not possible to get more data the only thing I am interested in ...
0
votes
0answers
21 views

Difference of network between testing and training on the same dataset (No training and testing)

I was training and dense net model on emotion recognition on the sewa dataset. Therefore, at the end I have 2 outputs. One for arousal and the other for valence (These dimensions for emotions). So I ...
0
votes
1answer
27 views

Can overfit happen in spite of validation and what to do with it?

Let's consider a standard situation where we need to find a predictive model. We train all the available model using a training data set. We validate all the trained model using a validation data ...
0
votes
1answer
16 views

How does train-validation-test procedure deals with the sampling error of the accuracy measure?

Let's consider a standard model selection procedure: We have N different untrained models (for example linear regression, neural network, decision tree and so on). We use a data set A to train each ...
0
votes
0answers
39 views

High dimensional regression overfitting

Consider the linear regression model \begin{equation} \boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{\epsilon} \end{equation} where we assume $\boldsymbol{X}$ is $n$-by-$p$, with $p &...
0
votes
0answers
28 views

Why do we take `(Bias) ^2` in total error in a model? [duplicate]

I was recently studying some book and few blogs and come to note that : Total error = Bias^2 +Variance + irreducible error Also, I know that these are the errors ...