Questions tagged [generalization-error]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
3
votes
2answers
154 views

What is accepted practice for avoiding optimistic bias when selecting a model family after hyperparameter tuning?

This is an extension of a previous question: How to avoid overfitting bias when both hyperparameter tuning and model selecting? ...which provided some options for the question at hand, but now I would ...
6
votes
2answers
994 views

How to avoid overfitting bias when both hyperparameter tuning and model selecting?

Say I have 4 or more algorithm types (logistic, random forest, neural net, svm, etc) each of which I want to try out on my dataset, and each of which I need to tune hyperparameters on. I would ...
7
votes
1answer
168 views

AIC model averaging when models are correlated

AIC model-averaging: In "standard" AIC model averaging we average models with weights proportional to $$w_i \propto \exp( -0.5 \times \Delta \text{AIC}_i ),$$ where $\Delta \text{AIC}_i$ is ...
0
votes
2answers
111 views

Is it Valid to Grid Search Cross Validation for Model Hyperparameter Selection then a separate Cross Validation for Generalisation Error?

The question has to do with Model Selection and Evaluation I'm trying to wrap my head around the scale of how different nested cross validation would be from the following: Let's say I am attempting ...
1
vote
1answer
51 views

Why do neural networks outperform SVMs on image recognition if SVMs have the less generalization error?

Why do neural networks outperform SVMs if SVMs have the less generalization error according to Vapnik? Is generalization error only useful in data scarce environments? Is it because neural networks ...
5
votes
1answer
76 views

what does it mean that there is leakage of information when one uses a test set?

I have read about the term "leakage of information" that occurs when one tries to estimate the generalization error by using a test set in Machine Learning models. However, I was not able to ...
1
vote
0answers
14 views

Model performance metric on the test sample

Since usually k-fold cross validation is carried out on the training sample I understand how the mean and the standard deviation of a metric are computed for the training sample but how is the mean ...
1
vote
0answers
69 views

Classification with noisy labels, noise is structured and not random

I am building a classification model with mislabeled training data on the order of ~70% of the training data is labeled correctly and ~30% is labeled incorrectly. Knowing this, how can I quantify the ...