5
The difference between the two should not be large. That said, Kim (2008) "Estimating classification error rate: Repeated cross-validation,repeated hold-out and bootstrap" that does present an investigation of repeated CV stipulates explicitly: "we obtain the $10$-fold CV estimates $5$ times, and take the average as the final estimate" when presenting the ...
4
@cbeleites actually to add on the following statement "But you need to be aware that this is a non-standard procedure and should be explained clearly, and the averages should IMHO ", there is the following paper which states:
"The problem with AUCmerge is that by sorting different folds together, it assumes that the classifier should produce well-...
3
It's for the same reasons you use cross-validation in any situation, as opposed to a single train/test split. You're able to leverage the entire dataset for both training and testing, which provides more robust performance estimates (as it's calculated over a larger N), and protects against "unlucky" train/test splits (which becomes more likely with smaller ...
3
There is no disadvantage in doing repeated CV in comparison with a single CV fold. If anything, repeated CV should decrease the variance of our estimate.
An excellent and highly-cited overview on cross-validation procedures can be found in Arlot & Celisse (2010) A survey of cross-validation procedures for model selection. The paper is admittedly a bit ...
3
The question is quite broad, but I will give some starting points:
Why bother with AIC/BIC: using cross validation (CV) is (much) more computationally expensive than using AIC/BIC, except for some special cases like leave-one-out cross validation (LOOCV) for regression where it is computationally as cheap as AIC/BIC.
Situations where AIC/BIC would not ...
2
In addition to @DemetriPananos' very good advise to look into ESL:
I find fig. 1 in Dietterich, T. G. (1998). Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10(7), 1895–1923. extremely helpful in clarifying what one is talking about (including $\mathrm Err_\tau$ vs. $\mathrm Err$).
If you are ...
answered Feb 20 at 16:33
cbeleites supports Monica
29k2 gold badges60 silver badges117 bronze badges
2
Intuitively, I'd like to get some estimate of how much the performance of these models varies depending on the sample I take.
Take a look at cross validation schemes. There are a couple interpretations of error in Elements of statistical learning which I think would be beneficial for you to know.
The first is Generalization or Test Error. This is the ...
2
In plain English, cross validated error is your best guess for the average error you would see with your regression model on new data.
I'm being a bit fast and loose. Plain English definitions sometimes sacrifice precision for utility. There are lots of "average errors" for which this could refer to: average error keeping the training the same? Average ...
1
A single designated test set is very common in the machine learning literature (see e.g. all these leaderboards on paperswithcode.com - I would guess they would [nearly?] all be a single test set). That's also what is done in kaggle competitions.
There are some arguments for and against this. E.g. if you did cross-validation you should ensure that every ...
1
Overfitting, and generalization, are quite different in reinforcement learning than in supervised learning. There's perhaps a joke to be made that statisticians fit to the training set, machine learning people fit to the test set and reinforcement learning people overfit to the test set. However, this is not quite fair.
In RL, you are trying to approximate ...
1
IMHO your confusion is spot on.
The difficulty arises from the fact that we mix variance from two different sources here into one standard deviation (or variance).
The number of independent tested cases determines how certain we can be about the performance of one particular surrogate model.
I'll refer to this variance as $\sigma^2_n$.
But there may be ...
answered Feb 10 at 13:13
cbeleites supports Monica
29k2 gold badges60 silver badges117 bronze badges
1
For tree based model, it can automatically handle redundant features, i.e. less useful features will not be selected as a split point. So you do not need to manually handle feature selection problems.
In many implementations of random forest or tree based boosting, the algorithm will automatically select a subset of features to build each tree. Therefore, ...
1
Is there a correct way / order to do [two kind of hyperparameter optimization]?
Yes: unless you know for sure that the different hyperparameters do not interact, they should to be optimized together.
Here, they do interact => optimize together
You can also optimize sequentially, but that should then become an iterative procedure:
optimize one type of ...
answered Feb 23 at 10:18
cbeleites supports Monica
29k2 gold badges60 silver badges117 bronze badges
1
The usual and most general formulation of cross validation is embarassingly parallel. Computational time is approximately $k$ times the time to train the model on the whole data set.
The time to train any such surrogate model may be sensitive to
the numeber of cases actually in training ($\frac{k-1}{k}$th of the available cases)
predicting the remaining $\...
1
Okay, cross-validation is a great starting point for hyperparameter tuning. There is no one right answer about this, but here are some general thoughts about your methods:
1. Train on the full train/validation dataset and use the test set as "new" validation.
I'm assuming this means that you train on the best hyperparameters, and test the resultant model ...
1
If you want to include the testing results on a data set whose test results were used for optimization (your "validation" set), I'd recommend to do that in a third, separate lift curve (same for any other figure of merit or diagram).
Yes, optimization is part of the model training and the model did learn from these results. We expect the optimization lift ...
answered Feb 14 at 10:03
cbeleites supports Monica
29k2 gold badges60 silver badges117 bronze badges
1
I'd say that the default should be to treat preprocessing as part of model training, i.e. do this inside the cross validation loop.
You can save computation in some cases by "pulling" the transformation before the cross validation. This is allowed as long as the transformation does not violate statistical independence. So transformations that involve each ...
answered Feb 13 at 15:21
cbeleites supports Monica
29k2 gold badges60 silver badges117 bronze badges
1
I switched to 10 folds which I thought might make a difference but alas no, it's hardly effected my overfitting.
With choosing the apparently best set of hyperparameters, choice of $k$ basically shouldn't affect the model.
I then decided to try 10x10 folds which has helped somewhat but not enough.
This is either spurious (see above) or points to the ...
1
The simplest thing to try is using stratification. In the case of $k$-fold cross-validation this will ensure that, each fold will have (approximately) the same percentage of samples for each class as in the original sample.
A somewhat more involved solution would be to use over-sampling, under-sampling or a synthetic sample generation procedure like SMOTE ...
1
If I have a data set of 500 observations, should i divide it in a train and test set with for example 375 (75%) train observations and 125 (25%) test observations, and perform cross-validation on the train set?
Yes, you should do it as the initial step, regardless of the situation (test_size=0.2 is also a reasonable default in sklearn).
Or should I ...
1
Deviance residuals shouldn't necessarily be normally distributed, even when everything is perfectly fine. So they needn't match a normal / follow a straight line on a qq-plot when that plot is based on a normal distribution. Although I use a logistic regression model as my example instead of Poisson, it may help you to read my answer here: Interpretation ...
answered Feb 6 at 19:57
1
You're right: Boxplots are a good way to show data, but when there are few data it is recommended to just plot the actual data themselves. Further, you are most interested in showing the relationship within the pairings (probably the difference, but possibly their ratio, etc.), not the raw data, although they could be plotted in the background. For a ...
answered Feb 6 at 16:58
1
With regards to the question of disadvantage, I think that needs refining.
Disadvantage compared to k-fold CV? For large samples, it's computational time (as you noted). For small samples, there is no apparent disadvantage for repeated k-fold CV.
Disadvantage compared to bootstrapping? For small samples, bootstrapping can be better at choosing between ...
Only top voted, non community-wiki answers of a minimum length are eligible
