Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Modeling error (especially sampling error) instead of replicable and informative relationships among variables improves model fit statistics, but reduces parsimony, and worsens explanatory and predictive validity.

2
votes
1answer
166 views

Can it be over fitting when validation loss and validation accuracy is both increasing?

Training a simple neural network over a very sparse matrix (Has 2400 features and 18000 train rows) for a binary classification problem. At the end of 1st epoch validation loss started to increase, ...
1
vote
2answers
41 views

Techniques to avoid overfitting

I have heard of several techniques to avoid overfitting: Validation curve: which let us choose the set of parameter with the minimum step between validation score and training score. But it seems ...
0
votes
1answer
16 views

Why does cross entropy loss for validation dataset deteriorate far more than validation accuracy when a CNN is overfitting?

I have noticed that the cross-entropy loss for validation dataset deteriorates after a certain number of epochs when training CNN's or MLP's. This is, of course, the sign that the network is ...
3
votes
5answers
359 views

Is using both training and test sets for hyperparameter tuning overfitting?

You have a training and a test set. You combine them and do something like GridSearch to decide the hyperparameters of the model. Then, you fit a model on the training set using these hyperparameters, ...
7
votes
3answers
808 views

Over fitting on purpose

Would it make sense to overfit a model on purpose? Say I have a use case where I know the data will not vary much respect to the training data. I'm thinking here about traffic prediction, where the ...
0
votes
0answers
13 views

A good performance with n-fold cross validation and overfiting

Is it possible to train a model (for regression or classification) that has overfited bu that that has a good performance in a n-fold cross validation?
0
votes
0answers
22 views

Overfitting, cross validation and validation curve

Some time ago i used to choose my hyperparameters only relying on the Test score returned by cross validation. But after reading How does cross-validation overcome the overfitting problem? i doubted ...
1
vote
0answers
23 views

What metrics to look at when experimenting with neural network hyperparameters?

So with other machine learning techniques I generally only look at the validation error when deciding on certain hyperparameters. I've been reading up on neural networks and it seems that hand tuning ...
0
votes
0answers
16 views

How to train a neural network after the hyperparameters have been found? What is the stopping criteria?

Say for example i use validation score for early stopping to prevent the network from overfitting and find the best hyperparameters using the CV techniques. Now that i have found the best ...
0
votes
1answer
20 views

Does not being able to overfit a single training sample mean that the neural network architecure or implementation is wrong? [duplicate]

Is the following hypothesis true ? If a simple neural network cannot overfit a single training sample, there is something wrong with its architecture or its implementation. To give you more ...
1
vote
1answer
38 views

Unstable Prediction Probabilities

We have a data set of a company that needs to predict their employee resignation status. So We developed four classification models "Bagging","Boosting", "RandomForest" and "Logistic". The task we ...
0
votes
1answer
30 views

why does data external from training and testing my neural network perform much worse than statistical accuracy?

I've got a problem with my neural network (used to recognize audio signals, an expansion of the UrbanSound dataset problem): when I fit the model the accuracy of both train and validation is near 90%. ...
0
votes
3answers
68 views

How to find out if a model is overfitted? [duplicate]

I have built 2 models: 1) precision: 0.80 - AUC ROC: 0.69 2) precision: 0.90 - AUC ROC: 0.94 I have posted both them to Kaggle as Titanic competition, the first model scored 0.7 and the second one ...
1
vote
1answer
57 views

when is it possible to have OLS fits better than random forest and LASSO?

I ran several different models on a mini data set of about 100 observations with 90 features. When I tried OLS with backward selection the model is significant with many features significant (82 ...
2
votes
0answers
75 views

RNN LSTM overfitting

I'm trying to build a dynamic RNN network for 2-class classification, and I just can't get rid of the overfitting. I have 5500 samples of class A, and 8000 for class B (total 13500). From that I take ...
2
votes
1answer
26 views

Validation set early stopping on custom metric

I am wondering whether it is ok to monitor validation set performance using a metric which is not optimized by the training algorithm, but which makes more sense in your domain. As a concrete example,...
0
votes
0answers
18 views

FineTuned VGG16 achieving great results on epoch 1, is this normal?

I'm training a model to classify whether a person is smiling (showing teeth) or not. I'm using Keras and I trained a VGG16 model loaded with the ImageNet weights, froze the first 4 layers and added a ...
2
votes
1answer
33 views

How to do k-fold cross validation with classifiers?

I want to cross-validate a model that plays the card game below (see image). I trained the model on a dataset of 1000 games, with the goal to maximise the profit of each game. It works great on the ...
1
vote
1answer
35 views

Why ensemble of many deep-learning models did not work?

I am trying to solve an image classification problem using DL, Keras and tensorflow. I added several layers of conv2D followed by batchnorm, pooling and dropout. I get a good accuracy ~95% with this. ...
0
votes
1answer
20 views

Text classification - regularization worsens validation score

I have the text classification problem. Dataset is imbalanced in terms of classes. I'm using StratifiedKFold and balanced weights updating during training LogisticRegression. Let's say my score is: ...
1
vote
0answers
19 views

Avoiding OCR performance coupling to upstream Bounding Box model

I have a model pipeline where I first use an object detection deep learning model to locate text regions in images of natural scenery (i.e. outdoor images), and then send the cropped region to a deep ...
1
vote
0answers
19 views

deep learning, overfitting and identification problem

I have build a convolutional network with around "only" 1900 parameters for 4600 images in training set (observations) and I am still overfitting the training set. I view the problem like a system of ...
0
votes
1answer
66 views

Why is cross validation error high upon overfitting?

http://www.cs.cornell.edu/courses/cs4780/2015fa/web/lecturenotes/lecturenote13.html ref: Figure 1: overfitting and underfitting Shouldn't cross validation error follow training error and remain low ?...
0
votes
0answers
4 views

Evaluation for embeddings using an overfitting linear classifier

I currently work on an algorithm that produces low dimensional latent representations or vertices of a graph (embeddings). To evaluate the quality of these embeddings, I use them on a classification ...
1
vote
1answer
35 views

Why is dropout causing my network to overfit so badly? [closed]

I've been experimenting with various simple neural networks to test their performance. When I use the following architecture, I'm getting some very bad test error, which looks like overfitting. $$\...
3
votes
1answer
443 views

A huge gap between training and validation accuracy, confusion with the concept of Overfitting

I have a fairly small dataset with 100 examples per class and 12 classes in total. Out of all the CNN models I have tried, the only inference I could make is that my training accuracy plateaus at 97%, ...
0
votes
2answers
37 views

Is this linear model overfitting when I add more parameters?

I am trying to figure out if my models are overfitting. This is a trend I noticed with my actual dataset associating metadata with compositional data. The more parameters I add, the better the ...
2
votes
1answer
40 views

Why don't GAN generators vastly overfit?

It seems that, if the GAN generator is simply mapping noise to a value which should be as indistinguishable as possible for the discriminator from the real data, the generator could simply learn to ...
1
vote
1answer
55 views

Generate music using LSTM using language model overfitting problem

I want to try generate music using LSTMs from MIDI data. The model is based on the prediction of the next notes based on the previous sequence - based on known language models eg. char-rnn. To train I ...
1
vote
1answer
22 views

What explanatory variables to exclude?

I am conducting research on cross-sectional data of ebay auctions and want to determine the effect of reputation on price. ebay offers several measures of reputation: a users "feedback score" (...
-1
votes
1answer
47 views

How to judge whether model is overfitting or not [duplicate]

I am doing video classification with a model combining CNN and LSTM. In the training data, the accuracy rate is 100%, but the accuracy rate of the test data is not so good. The number of training data ...
0
votes
1answer
20 views

Unbalance images dataset

I want to create a deep learning model to classify images. My dataset has around 400 classes and the classes have different number of images.. How can I train the deep learning network on unbalanced ...
0
votes
0answers
53 views

Should feature selection be done on the same data that you train your model on?

I know that you should separate your data into training and validation sets before doing feature selection, to avoid getting too good, false results on cross validation. But, I have seen people say ...
1
vote
1answer
36 views

Can you overfit with proper scoring rules, e.g., Brier score?

I have read a lot suggestions and literature about using Brier score to measure model performance. It seems to be likened to the holy grail of model evaluation metrics because it is a proper scoring ...
1
vote
1answer
197 views

How to know if model is overfitting or underfitting?

I understand that using cross validation we can validate our model, but it is also possible that maybe our model is underfitting; hence, providing wrong results. One possibility that I can think of is ...
-1
votes
1answer
30 views

Why fewer feature classification models can perform better?

I have seen articles on the “curse of dimensionality” and why reducing the number of features can help with overfitting, but imagine we are interested in cases with significantly less features. Why is ...
0
votes
0answers
10 views

Deep VBPR Regularization Term question

I'm trying to understand/run the code in the repo below: https://github.com/kang205/DVBPR It's a tensorflow implementation in python of the model described in this paper: "Visually-Aware Fashion ...
1
vote
1answer
35 views

when do i know i am overfitting a model?

This example comes from the documentation of Matlab. Suppose you have data points ...
0
votes
0answers
33 views

MCMC/Bayesian Inference Model Fitting: Parameters over-fitting to a sharply peaked data point. How can I fix this?

I am trying to fit three models to a data set: Method: Each measurement has some measurement uncertainty Generate X samples from a gaussian with mean of the measurement and std of the measurement ...
2
votes
0answers
43 views

Is there a method to update support vector machine (SVM) parameters?

Consider, there are two classes of data and we have learned the SVM parameters in terms Lagrange multipliers. There are many learning techniques to learn these parameters quadratic programming or ...
3
votes
4answers
137 views

Why don't we want to choose a big $\lambda$ in ridge regression?

The author in this video at minute 16:15 says that: we don't want to choose big $\lambda$ values becuase the coefficients will become very small and therefore they might not be accurately ...
1
vote
1answer
318 views

Dealing with LSTM overfitting

I'm carrying out a project of predicting time series data with an LSTM. I tried out the experiment three times with randomly sampled data(about 920,000 lines each) I've stacked 3 layers of LSTM cells,...
0
votes
1answer
36 views

Is the solution at tangent point an optimal solution?

From what I understood from this article, the blue circles are the level curves and the blue dot is the optimal solution that minimizes the cost function. The yellow circle is the L2-norm constraint. ...
1
vote
1answer
54 views

Why limiting the weights to not grow to big numbers help to avoid overfitting? [duplicate]

I understand that in order to avoid overfitting we need to reduce the complexity of the network. Or, in other words we can reduce the degree of polynomial. L1-norm does exactly this - reduces the ...
4
votes
2answers
205 views

What is the geometric explanation for high variance = overfitting = bad generalization?

I am reading about the L2 regularization. According to Python Machine Learning - Second Edition, "by increasing the regularization strength via the regularization parameter λ , we shrink the weights ...
0
votes
1answer
18 views

How to choose between two models having similar validation score but different training score?

I have trained two models, A and B. A has more parameters than B. Also A has higher training score than B. However both models have similar validation scores. Obviously model A overfits. How do I ...
1
vote
0answers
171 views

Text classification using tf idf

I am developing a text classification model. At this moment I have to classify some documents in six different classes. I am using a simple approach as a starting point based on random forests over a ...
1
vote
1answer
26 views

Overfitting and data transformations

I've read a lot about overfitting here, and now I have a question about this subject. Ok, if we put together too much independent variables, primary looking for better fits (for exemple, higher R², ...
7
votes
2answers
261 views

Is Cross Validation useless unless the Hypotheses are nested?

If I generate many random models (without considering the data at all) in a regression setting simply by randomly assigning coefficient values and then evaluating these models over the dataset with an ...
5
votes
1answer
268 views

Higher overfitting using data augmentation with noise?

I am training a neural network for Audio classification. I trained it on the UrbanSound8K dataset (Model1), and then I wanted to evaluate how different levels of added noise to the inputs influenced ...