Questions tagged [overfitting]

Modeling error (especially sampling error) instead of replicable and informative relationships among variables improves model fit statistics, but reduces parsimony, and worsens explanatory and predictive validity.

244 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
9
votes
0answers
442 views

Can Frank Harrell's method be used to obtain optimism-corrected regression coefficients?

I used a regularized (LASSO) Cox regression to estimate relapse times of patients and used Frank Harrell's bootstrapping method to obtain an optimism-corrected performance estimate of my model. ...
6
votes
1answer
459 views

How do bias, variance and overfitting relate to each other?

I'm quite new to Machine Learning, and after reading about the bias-variance tradeoff and overfitting/underfitting, several questions raised in my mind: If I have a model with 15% error on train set ...
6
votes
0answers
899 views

Signatures of underfitting and overfitting in logistic regression calibration curves

My confusion stems from reading the following paper http://www.bmj.com/content/351/bmj.h3868 It states in its abstract (and they later show an empirical study that conforms to the claim) - "...
6
votes
2answers
1k views

Gaussian Mixture Model - Model selection using the held-out likelihood?

I am trying to understand how to select the number of components in a Gaussian Mixture Model (GMM). Most presentations mention the use of criteria such as AIC and BIC. But if we simply follow model ...
5
votes
1answer
466 views

Measuring the bias-variance tradeoff

Does anyone know of a metric that quantifies the bias-variance tradeoff of a given fitted model? I'm not talking about measuring the MSE in cross validation, I'm interested in a single generic or ...
4
votes
0answers
76 views

Predicting a Twitter user from individual tweets probabilities

Let's say I have three tweets and those three tweets are all from either Mary or John. There is no possibility for mixed result. ...
4
votes
0answers
176 views

Rigorous theory behind overfitting

I am taking an intro to ML class, and in my limited experience, training ML algorithms (validation, overfitting etc.) feels a bit like black magic. For instance, you aren't supposed to touch the test ...
3
votes
0answers
33 views

How can validation loss increases than slowly decreases when overfitting?

I'm using CNN to do multi-regression jobs, but some wired results confuse me. Here's my training results curves: As you can see, the training loss first decreases fastly and then converges, while ...
3
votes
0answers
86 views

Over “specification” of a statistics model?

For a simple example, I am fitting data with a likelihood function generated by a normal distribution. The first model is the normal distribution with two-parameters. The second competing model is the ...
3
votes
0answers
897 views

Overfitting in Random Forest Classifier?

I would like some help from you in a classification model that I am developing. In summary, the problem is: – Classification problem with binary outcome (0/1) – The classifier is a Random Forest ...
3
votes
0answers
358 views

Understanding KS Statistic as Model Selection Tool

As a hobbyist learning about predictive modeling and machine learning, I am having some difficulty finding clarity regarding the KS statistic as a method for model selection. My mentor has been ...
3
votes
2answers
41 views

Is it problematic if most of the weights or biases in a hidden layer are the same sign?

I'm trying to diagnose overfitting in my multi-layer perceptron by looking at the weights, biases and gradients in each layer. I'm noticing that in the neural network that is overfitting, the weights ...
3
votes
0answers
342 views

Why does RNN overfit for sentiment analysis but not for spam detection?

I used this code which uses RNNs for spam detection and got reasonable results. But when I use the same code for sentiment analysis, the model overfits badly: its training accuracy keep growing, but ...
3
votes
0answers
131 views

Why do Srivastava et al. claim that “the best” theoretical regularization technique involves all possible network parameter settings?

In the original paper on Dropout by Srivastava, Hinton, Krizhevsky et al. (2014), the authors make this claim in the introduction: With unlimited computation, the best way to "regularize" a fixed-...
3
votes
1answer
673 views

Train accuracy < Test accuracy with regularization

With a friend we were playing with the notMNIST data, logistic regression and regularization. Without regularization, we could achieve a training accuracy (10k samples) of 78%, and test accuracy (15k ...
3
votes
0answers
62 views

Does this pattern indicate over-fitting in machine learning?

I am working on a diagnostics project, and trying to improve the performance of a classifier(s). We have over a million features to choose from, so feature selection is a real challenge. To look ...
3
votes
0answers
166 views

How to avoid an overfitting?

The standard way to avoid an over-fitting is to use a "validation set". It means that we split the data into two parts. The first part we use to fit (train) and the second part we use to validate. ...
3
votes
0answers
148 views

Are regression problems more likely to overfit than classification problems?

I will illustrate my question on an example: Let's say we have a dataset that we want to split into two disjoint sets of similar size. The dataset has a high dimensional feature (several hundred ...
3
votes
0answers
404 views

Is approximating used car prices with deep learning over-engineering?

I am supposed to build an application that uses deep learning to approximate prices of used cars. My concern is that deep learning is too general of a tool for the problem at hand. I am going to use ...
3
votes
0answers
400 views

LASSO prediction model question

I am trying to create a prediction model with 33 predictors (brain metabolite levels in various regions) and 8 observations (cognitive test scores) with p>>n problem using LASSO in MATLAB (...
3
votes
0answers
253 views

How to select the exponential decay constant for weighting in proc logistic?

I am trying to predict the sales conversion using proc logistics in SAS. Right now I have around 3 months of data, and it will gradually grow to more than an year over time. My intuition is that the ...
2
votes
1answer
63 views

Is there a better way to describe a model's generalization performance than “under” and “overfitting”?

To me, under and overfitting are the two of the most vague concepts in machine learning. From Google's first link when you look up these definitions. A model is said to be underfitted if it "...
2
votes
1answer
25 views

Is it possible to overfitting within single epoch

Let me put my question first. For a time-series prediciton, is it possible to overfit even within the first epoch, when training data and validation data should all "new" to model? Features and ...
2
votes
0answers
31 views

Removing duplicate records or not when using FastText

I am currently working on a project classifying text into classes. The specific problem is classifying job titles into various industry codes. For example "McDonalds Employee" might get classified to ...
2
votes
0answers
21 views

What are the available method that can alleviate the overfitting problem in traditional OLS problem, but still can get a linear fitting?

Recently, I have read the paper https://static1.squarespace.com/static/56def54a45bf21f27e160072/t/5a0d0673419202ef1b2259f2/1510803060244/The_Sampling_Error_in_Estimates_of_Mean-...
2
votes
1answer
186 views

how to avoid overfittig with xgboost and how to increase accuracy

I am doing a binary classification problem, I got to train 85% accuracy, but test accuracy is 72%, I tried different parameters, Cross valid, But overfitting doesn't change, please help me on how to ...
2
votes
1answer
192 views

Can logistic regression output a non linear curve?

I have a doubt regarding logistic regression.I know that it separates the data into 2 parts.Is it possible that it leads to a curve as shown in the example of underfitting and overfitting What i know ...
2
votes
1answer
39 views

Fitting data with unknown noise with complicated model — can anything be done?

I have some experimental data, where due to the nature of the experiment (which I am not familiar with) the errors are not known. The data follow a rough non-linear trend but clearly are noisy as for ...
2
votes
0answers
162 views

Overfitting cross-validation scores

I've ran 1000 iterations of XGBClassifier parameters search using RandomizedSearchCV on the titanic dataset. That's just for context, but the question applies to any CV search method, any model and ...
2
votes
1answer
846 views

Finding the appropriate polynomial fit in Python

Is there a function or library in Python to automatically compute the best polynomial fit for a set of data points? I am not really interested in the ML use case of generalizing to a set of new data, ...
2
votes
1answer
49 views

With test accuracy being equal, is it better to have lower training accuracy?

Suppose we train two models on a training set, and then test them both on the training set itself, and on a test set. We have some accuracy metric we're using to evaluate them. Both models score ...
2
votes
0answers
188 views

Bias and over-fitting in Maximum Likelihood estimation

In his book, "Pattern recognition and Machine learning", Bishop talks about the influence of the bias and overfitting in the MLE framework. Here is a quote from p.28, just before he has shown that the ...
2
votes
0answers
2k views

RNN LSTM overfitting

I'm trying to build a dynamic RNN network for 2-class classification, and I just can't get rid of the overfitting. I have 5500 samples of class A, and 8000 for class B (total 13500). From that I take ...
2
votes
0answers
90 views

Avoiding OCR performance coupling to upstream Bounding Box model

I have a model pipeline where I first use an object detection deep learning model to locate text regions in images of natural scenery (i.e. outdoor images), and then send the cropped region to a deep ...
2
votes
1answer
53 views

when do i know i am overfitting a model?

This example comes from the documentation of Matlab. Suppose you have data points ...
2
votes
0answers
83 views

Is there a method to update support vector machine (SVM) parameters?

Consider, there are two classes of data and we have learned the SVM parameters in terms Lagrange multipliers. There are many learning techniques to learn these parameters quadratic programming or ...
2
votes
0answers
71 views

What is the relation between replica method and “reusable holdout” method?

Among many methods used to detect and avoid overfitting, I am particularly interested in those two: replica method reusable holdout My question is: what is their relation in the context of adaptive ...
2
votes
1answer
150 views

Higher Test Scores but Higher Variance?

I am tuning hyper-parameters using 5-fold cross-validated grid search for various multiclass classifiers, and I keep running into the same issue that I can't quite wrap my head around. The hyper-...
2
votes
1answer
311 views

Why do increasing regularization weights make objective function not monotonically decrease?

I run modified non-negative matrix factorization (NMF) and tune the regularization weight from 1e5 to 1e13. The table below ...
2
votes
0answers
280 views

Does minimum norm solution guarantee generalization in the underconstrained case (in the statistical learning sense)?

Recall that pseudo-inverse can be characterized as follows: Solve $$ \| w \|^2 $$ subject to: $$ Xw = y $$ thus it is plausible since its a constrained optimization problem that the solution ...
2
votes
0answers
493 views

General strategies to avoid overfitting GAMs with very little noise

I have been working with GAMs (R package 'mgcv') and SCAMs (R package 'scam'), applied to some simulated data I have generated (rarefaction curves for ecological analysis). What I am finding is that ...
2
votes
0answers
97 views

Bayesian Model Suddenly Overfits

I am building a Bayesian model and have a trouble of (sudden) overfitting. The example figure that shows the issue is here: Until around 1250 iteration, the log-likelihood goes up little by little, ...
2
votes
1answer
2k views

LSTM extreme overfitting on learning rate reduction

I'm applying a single layer LSTM with hidden_size=16 towards a binary classification task. My training and validation loss are both reasonable until around epoch 400 when my learning rate gets halved, ...
2
votes
0answers
97 views

Applying machine learning to dynamic complex systems (e.g. weather prediction)

Would it be correct to say that: Physics-based, domain specific models are more widely used and are more practical in (longer term) weather forecasts than pure machine learning approaches The reason ...
2
votes
0answers
115 views

Gradient boosting for regression

I was going through this this as an introduction to gradient boosting to get an overview. The algorithm described for regression is Let n be number of samples and be the response variable. and ...
2
votes
0answers
399 views

I have p >> n and no overfitting, why?

I try to classify a brainstate (binary problem) on fMRI-data using a SVM (scikit-learn, which wrapps libsvm). Also I use clusters arround local maxima in group-level TMaps as mask for the subject ...
2
votes
0answers
728 views

Demonstrating Overfitting in a Simple Model

I have been working with a finance team to help forecast revenue for some product data. Particularly when the series are short and difficult to forecast, their first response is to add a bunch of "...
2
votes
0answers
359 views

why k-fold cross validation (CV) overfits? Or why discrepancy occurs between CV and test set?

Recently I am working on a project and I found my cross-validation error rate very low but the testing set error rate is high, which might indicate the overfiting of my model. But why my cross-...
2
votes
0answers
149 views

Why if we have a singular covariance matrix then the data is over fitted?

If we have D dimensional training data and the covariance matrix was not exist for those data, then we have over fitting. I think this is related to make sure that the observations must has low ...
2
votes
0answers
497 views

Optimization vs. prediction methods - train/test sets needed?

Let's say you are a city builder and want to optimize the number of laundromats and convenience stores based on the satisfaction of people (just a made-up example). Your data might look like this: <...

1
2 3 4 5