Questions tagged [generalization-error]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
12 views

Generalization error of linearly predicting a random smooth function from $D$ independent random smooth functions?

Suppose I have training data $\{x_p\}_{p=1}^P$ of $P$ patterns, with $x_p \sim_{i.i.d.} p(x)$ and I do the following: Draw $D$ random functions $f_1, ..., f_D \sim_{ i.i.d.} \mathcal{GP}(0, k_{\sigma}...
user avatar
0 votes
0 answers
13 views

After training a model, how does test set error inform decision making?

I split a data set into three subsets: training, validation, and test sets. I use my training data for fitting and validation to check for overfitting. I then have a final model that I then propose to ...
user avatar
  • 1,093
3 votes
2 answers
200 views

What is accepted practice for avoiding optimistic bias when selecting a model family after hyperparameter tuning?

This is an extension of a previous question: How to avoid overfitting bias when both hyperparameter tuning and model selecting? ...which provided some options for the question at hand, but now I would ...
user avatar
  • 226
8 votes
2 answers
2k views

How to avoid overfitting bias when both hyperparameter tuning and model selecting?

Say I have 4 or more algorithm types (logistic, random forest, neural net, svm, etc) each of which I want to try out on my dataset, and each of which I need to tune hyperparameters on. I would ...
user avatar
  • 226
8 votes
1 answer
276 views

AIC model averaging when models are correlated

AIC model-averaging: In "standard" AIC model averaging we average models with weights proportional to $$w_i \propto \exp( -0.5 \times \Delta \text{AIC}_i ),$$ where $\Delta \text{AIC}_i$ is ...
user avatar
  • 23.4k
0 votes
2 answers
155 views

Is it Valid to Grid Search Cross Validation for Model Hyperparameter Selection then a separate Cross Validation for Generalisation Error?

The question has to do with Model Selection and Evaluation I'm trying to wrap my head around the scale of how different nested cross validation would be from the following: Let's say I am attempting ...
user avatar
1 vote
1 answer
51 views

Why do neural networks outperform SVMs on image recognition if SVMs have the less generalization error?

Why do neural networks outperform SVMs if SVMs have the less generalization error according to Vapnik? Is generalization error only useful in data scarce environments? Is it because neural networks ...
user avatar
5 votes
1 answer
80 views

what does it mean that there is leakage of information when one uses a test set?

I have read about the term "leakage of information" that occurs when one tries to estimate the generalization error by using a test set in Machine Learning models. However, I was not able to ...
user avatar
  • 599
1 vote
0 answers
17 views

Model performance metric on the test sample

Since usually k-fold cross validation is carried out on the training sample I understand how the mean and the standard deviation of a metric are computed for the training sample but how is the mean ...
user avatar
1 vote
0 answers
85 views

Classification with noisy labels, noise is structured and not random

I am building a classification model with mislabeled training data on the order of ~70% of the training data is labeled correctly and ~30% is labeled incorrectly. Knowing this, how can I quantify the ...
user avatar
  • 1,408