Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

All Questions

3
votes
2answers
129 views

What's the real purpose of cross validation?

As for cross evaluation (CV), I have two questions to ask: 1) CV has nothing to do with parameter selection, but only model evaluation? Specifically, which model? 2) In k-fold CV, what's the final ...
1
vote
0answers
45 views

What is the relation between replica method and “reusable holdout” method?

Among many methods used to detect and avoid overfitting, I am particularly interested in those two: replica method reusable holdout My question is: what is their relation in the context of adaptive ...
3
votes
0answers
277 views

Nested Cross Validation: Choosing between different best hyperparameters

I know this sort of question has been asked many times, and several answers have been already provided on this platform too (e.g., here, here, and here). Still, there is something about the idea ...
15
votes
1answer
762 views

Is Kaggle's private leaderboard a good predictor of out-of-sample performance of the winning model?

While the results of the private test set can not be used to refine the model further, isn't model selection out of a huge number of models being performed based on the private test set results? Would ...
2
votes
2answers
3k views

Pros and cons of cross-validation?

A practicing statistician I know advocates strongly against cross-validation, claiming that he would rather build the model on the entire dataset and make sure the underlying statistical assumptions ...
3
votes
1answer
454 views

Do I need an initial train/test split for nested cross-validation?

I have a couple of pipelines: pipeline 1: CV'd feature selection, CV'd hyperparameter selection for classifier A pipeline 2: CV'd feature selection, CV'd hyperparameter selection for classifier B ...
1
vote
1answer
491 views

Model selection and performance evaluation with different sample sizes

Suppose there are K experimental units. Each unit is associated with its own dataset consisting of 400 observations. For each unit, we set up a two-sample test, 200 vs 200. Because of a large sample ...
0
votes
1answer
59 views

Holdout set for image task

I need to validate whether one or two templates/shapes are present in an image. Fitting two templates has a better maximum likelihood then fitting one template which is a clear symptom of overfitting. ...
2
votes
1answer
160 views

Testing the variance part of a Generalized Linear Model out of sample

Suppose I have a response vector and a factorial design (for simplicity, assume it’s a one-way ANOVA with two treatments). A few Generalized Linear Models (Poisson, Negative Binomial, etc) are fitted ...
3
votes
1answer
1k views

Adjusted R squared on a holdout set

The formula for adjusted $R^2$ is: $$ 1 - \frac{(n-1)}{(n-p-1)}(1-R^2) $$ where $r^2$ is the coefficient of determination, $n$ is the number of points, and $p$ is the number of parameters the model ...