Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Modeling error (especially sampling error) instead of replicable and informative relationships among variables improves model fit statistics, but reduces parsimony, and worsens explanatory and predictive validity.

0
votes
0answers
15 views

Bias and over-fitting in Maximum Likelihood estimation

In his book, "Pattern recognition and Machine learning", Bishop talks about the influence of the bias and overfitting in the MLE framework. Here is a quote from p.28, just before he has shown that the ...
1
vote
0answers
14 views

Why, when we use the logarithmic function, weigh less to overfit? [on hold]

Why, when we use the logarithmic function, weigh less to overfit, or why I create less local logarithmic errors, they wanted an answer with proof.
0
votes
0answers
9 views

Why after multiple epoc the performance decrease on the train data?

I have a network and I notice that after multiple epoc the performance on the train data start decreasing. Why?
2
votes
3answers
78 views

Maximum likelihood estimators and overfitting

In his book, Bishop claims that overfitting is caused by an unfortunate property of the Maximum likelihood estimator. I dont really understand how the MLE relates to overfitting. To me, roughly, ...
2
votes
1answer
32 views

Why L1 regularization can “zero out the weights” and therefore leads to sparse models? [duplicate]

I'm aware there is a very relevant explanation on L1 regularization's effect on feature selection at here: Why L1 norm for sparse models [Ref. 1]. To better understand it I'm reading Google's ...
0
votes
1answer
18 views

Does dropout regularization prevent overfitting due to too many iterations?

For image classification problem, let's say, and given a neural network to train on, if you were to run too many iterations for a single image of a cat would not generalize well into other images of ...
0
votes
0answers
50 views

How to interpret/choose alpha in ridge regression

I have questions on how to apply ridge regression on my data set, which has about 75 samples with 8 features (x's) and usually 3 targets (y's). I tried the following feature engineering methods. ...
1
vote
1answer
24 views

Model overfitting when using two separate datasets for train and test

I have two datasets generated from two FPGA cricuits having almost same design. Both have 17 features as binary values where the last column is the class label <...
4
votes
1answer
38 views

Does Leave One Out cross validation increase the chance of overfitting?

By increasing the size of the training set the model memorize more data. Thus, will using leave one out increase the chance of overfitting?
1
vote
2answers
47 views

How to prevent overfitting in Gaussian Process

I'm training Gaussian Process models on a relatively small data set, which have 8 input features and 75 input data. I tried different kernels and find the following kernel (2 RBF + a white noise)...
0
votes
1answer
41 views

How to prevent overfitting with regression using ranger (randomforest)

I use caret to train the model (on Boston dataset from the mlbench package). Here is the code ...
2
votes
2answers
336 views

Can overfitting be a good thing in some cases?

I know the goal of machine learning is to create generalizable models and therefore overfitting is undesirable. However, I wonder if it could be desirable in some cases. For example, let's say I ...
0
votes
1answer
44 views

how to choose model when training accuracy is lower than validation accuracy while training neural network?

Below is a specific case but a general situation i find myself involved with in my job. This question is intended at getting ideas on how to pick the best model: Dataset: rows: 10,166, features: ...
1
vote
0answers
34 views

Bias-Variance terminology for loss functions in ML vs cross-validation — different things?

I am a bit confused about the use of variance and bias across the machine learning and statistical learning literature. In particular, the bias-variance trade-off arises from the fact that one can ...
0
votes
0answers
21 views

Obtaiting the Training Error from a RandomForestRegressionModel

I was planning to plot the evolution of my RandomForestRegressionModel when tunning the hyperparameters. One thing I would like to evaluate is the overfitting. For ...
0
votes
0answers
20 views

Comparing Model's Performance on the Train and Test Sets

I've developed a model that predicts a future value of a parameter for the next 72 hours (only 11 hours presented on the chart). I've obtained the hyperparameters for my model with use of ...
2
votes
1answer
236 views

Can it be over fitting when validation loss and validation accuracy is both increasing?

Training a simple neural network over a very sparse matrix (Has 2400 features and 18000 train rows) for a binary classification problem. At the end of 1st epoch validation loss started to increase, ...
1
vote
2answers
73 views

Techniques to avoid overfitting

I have heard of several techniques to avoid overfitting: Validation curve: which let us choose the set of parameter with the minimum step between validation score and training score. But it seems ...
0
votes
1answer
46 views

Why does cross entropy loss for validation dataset deteriorate far more than validation accuracy when a CNN is overfitting?

I have noticed that the cross-entropy loss for validation dataset deteriorates after a certain number of epochs when training CNN's or MLP's. This is, of course, the sign that the network is ...
3
votes
5answers
498 views

Is using both training and test sets for hyperparameter tuning overfitting?

You have a training and a test set. You combine them and do something like GridSearch to decide the hyperparameters of the model. Then, you fit a model on the training set using these hyperparameters, ...
8
votes
3answers
866 views

Overfitting on purpose

Would it make sense to overfit a model on purpose? Say I have a use case where I know the data will not vary much respect to the training data. I'm thinking here about traffic prediction, where the ...
0
votes
0answers
13 views

A good performance with n-fold cross validation and overfiting

Is it possible to train a model (for regression or classification) that has overfited bu that that has a good performance in a n-fold cross validation?
0
votes
0answers
34 views

Overfitting, cross validation and validation curve

Some time ago i used to choose my hyperparameters only relying on the Test score returned by cross validation. But after reading How does cross-validation overcome the overfitting problem? i doubted ...
1
vote
0answers
24 views

What metrics to look at when experimenting with neural network hyperparameters?

So with other machine learning techniques I generally only look at the validation error when deciding on certain hyperparameters. I've been reading up on neural networks and it seems that hand tuning ...
0
votes
0answers
18 views

How to train a neural network after the hyperparameters have been found? What is the stopping criteria?

Say for example i use validation score for early stopping to prevent the network from overfitting and find the best hyperparameters using the CV techniques. Now that i have found the best ...
0
votes
1answer
26 views

Does not being able to overfit a single training sample mean that the neural network architecure or implementation is wrong? [duplicate]

Is the following hypothesis true ? If a simple neural network cannot overfit a single training sample, there is something wrong with its architecture or its implementation. To give you more ...
1
vote
1answer
50 views

Unstable Prediction Probabilities

We have a data set of a company that needs to predict their employee resignation status. So We developed four classification models "Bagging","Boosting", "RandomForest" and "Logistic". The task we ...
0
votes
1answer
32 views

why does data external from training and testing my neural network perform much worse than statistical accuracy?

I've got a problem with my neural network (used to recognize audio signals, an expansion of the UrbanSound dataset problem): when I fit the model the accuracy of both train and validation is near 90%. ...
0
votes
3answers
125 views

How to find out if a model is overfitted? [duplicate]

I have built 2 models: 1) precision: 0.80 - AUC ROC: 0.69 2) precision: 0.90 - AUC ROC: 0.94 I have posted both them to Kaggle as Titanic competition, the first model scored 0.7 and the second one ...
1
vote
1answer
69 views

when is it possible to have OLS fits better than random forest and LASSO?

I ran several different models on a mini data set of about 100 observations with 90 features. When I tried OLS with backward selection the model is significant with many features significant (82 ...
2
votes
0answers
145 views

RNN LSTM overfitting

I'm trying to build a dynamic RNN network for 2-class classification, and I just can't get rid of the overfitting. I have 5500 samples of class A, and 8000 for class B (total 13500). From that I take ...
2
votes
1answer
72 views

Validation set early stopping on custom metric

I am wondering whether it is ok to monitor validation set performance using a metric which is not optimized by the training algorithm, but which makes more sense in your domain. As a concrete example,...
0
votes
0answers
19 views

FineTuned VGG16 achieving great results on epoch 1, is this normal?

I'm training a model to classify whether a person is smiling (showing teeth) or not. I'm using Keras and I trained a VGG16 model loaded with the ImageNet weights, froze the first 4 layers and added a ...
2
votes
1answer
47 views

How to do k-fold cross validation with classifiers?

I want to cross-validate a model that plays the card game below (see image). I trained the model on a dataset of 1000 games, with the goal to maximise the profit of each game. It works great on the ...
1
vote
1answer
42 views

Why ensemble of many deep-learning models did not work?

I am trying to solve an image classification problem using DL, Keras and tensorflow. I added several layers of conv2D followed by batchnorm, pooling and dropout. I get a good accuracy ~95% with this. ...
0
votes
1answer
23 views

Text classification - regularization worsens validation score

I have the text classification problem. Dataset is imbalanced in terms of classes. I'm using StratifiedKFold and balanced weights updating during training LogisticRegression. Let's say my score is: ...
1
vote
0answers
29 views

Avoiding OCR performance coupling to upstream Bounding Box model

I have a model pipeline where I first use an object detection deep learning model to locate text regions in images of natural scenery (i.e. outdoor images), and then send the cropped region to a deep ...
1
vote
0answers
19 views

deep learning, overfitting and identification problem

I have build a convolutional network with around "only" 1900 parameters for 4600 images in training set (observations) and I am still overfitting the training set. I view the problem like a system of ...
0
votes
1answer
342 views

Why is cross validation error high upon overfitting?

http://www.cs.cornell.edu/courses/cs4780/2015fa/web/lecturenotes/lecturenote13.html ref: Figure 1: overfitting and underfitting Shouldn't cross validation error follow training error and remain low ?...
0
votes
0answers
5 views

Evaluation for embeddings using an overfitting linear classifier

I currently work on an algorithm that produces low dimensional latent representations or vertices of a graph (embeddings). To evaluate the quality of these embeddings, I use them on a classification ...
1
vote
1answer
37 views

Why is dropout causing my network to overfit so badly? [closed]

I've been experimenting with various simple neural networks to test their performance. When I use the following architecture, I'm getting some very bad test error, which looks like overfitting. $$\...
4
votes
1answer
497 views

A huge gap between training and validation accuracy, confusion with the concept of Overfitting

I have a fairly small dataset with 100 examples per class and 12 classes in total. Out of all the CNN models I have tried, the only inference I could make is that my training accuracy plateaus at 97%, ...
0
votes
2answers
48 views

Is this linear model overfitting when I add more parameters?

I am trying to figure out if my models are overfitting. This is a trend I noticed with my actual dataset associating metadata with compositional data. The more parameters I add, the better the ...
2
votes
1answer
66 views

Why don't GAN generators vastly overfit?

It seems that, if the GAN generator is simply mapping noise to a value which should be as indistinguishable as possible for the discriminator from the real data, the generator could simply learn to ...
1
vote
1answer
67 views

Generate music using LSTM using language model overfitting problem

I want to try generate music using LSTMs from MIDI data. The model is based on the prediction of the next notes based on the previous sequence - based on known language models eg. char-rnn. To train I ...
1
vote
1answer
23 views

What explanatory variables to exclude?

I am conducting research on cross-sectional data of ebay auctions and want to determine the effect of reputation on price. ebay offers several measures of reputation: a users "feedback score" (...
-1
votes
1answer
54 views

How to judge whether model is overfitting or not [duplicate]

I am doing video classification with a model combining CNN and LSTM. In the training data, the accuracy rate is 100%, but the accuracy rate of the test data is not so good. The number of training data ...
0
votes
1answer
39 views

Unbalance images dataset

I want to create a deep learning model to classify images. My dataset has around 400 classes and the classes have different number of images.. How can I train the deep learning network on unbalanced ...
0
votes
0answers
55 views

Should feature selection be done on the same data that you train your model on?

I know that you should separate your data into training and validation sets before doing feature selection, to avoid getting too good, false results on cross validation. But, I have seen people say ...
1
vote
1answer
44 views

Can you overfit with proper scoring rules, e.g., Brier score?

I have read a lot suggestions and literature about using Brier score to measure model performance. It seems to be likened to the holy grail of model evaluation metrics because it is a proper scoring ...