3
1) Using 5 (or k) splits gives more information for finding your parameters than just using a single split. If your training sample has 500 observations, the first split evaluates the quality on fold 1, that's 100 observations. If you use all five, you can evaluate on 500 observations. Your skepticism is probably based on the fact that the data in the ...
3
The $R^2$ value is rounded to 3 decimal places. If the exact $R^2$ is $\ge$0.9995, it would then be reported as 1.000.
2
Can AIC and BIC be used on the left-out fold in a 10-fold cross-validation?
No, that would not make sense. AIC and cross validation (CV) offer estimates of the model's log-likelihood* of new, unseen data from the same population from which the current data sample has been drawn. They do it in two different ways.
AIC measures the log-likelihood of the ...
2
I think the key thing to note is that split 1, split 2 etc. are completely independent in terms of knowledge of model fit from one split to another (1). The model fits (training and test) have no way of knowing what happened in split 1, it forgets it all and starts again.
In split 1, Fold 1 are the test data, and Folds 2, 3, 4, and 5 the training data. ...
1
In the end, you pick a model which is estimated on the whole data. This is not a model which is a weighted combination of inferior models estimated on only a subset of the data (which you seem to be implying).
In the final, "optimal" model you need to have only statistically significant terms. For example, if the slope coefficient is non-significant ...
1
AFAIK, bootstrapping is for getting the standard error estimates. If you want to validate models, look at N-fold cross-validation, or leave-one-out crossvalidation (jacknife).
EDIT: since you said that years are not serially correlated, this simplifies your situation a lot! You can then treat the years as if they were just data points, and select the ...
1
Two compare two regression models (RM) using hypothesis testing two cases need to be considered:
Large data set S: One can divide S into several disjoints training sets and a single test set. Each RM is trained on each training set and then tested in the test set. An analysis of variance using the quasi-F test can be performed to test if RM1 is better than ...
1
I use the repeats as I have a small dataset (<200) and would like tighter bounds on my model performance for significance testing.
Repetitions of $k$-fold cross validation allow you to measure model instability, and the uncertainty related to that source of variance will go down if you average repetitions.
But it doesn't do anything about the actual ...
1
Writing an answer as answering in the comments is not really desirable.
Why are different models not just trained on the same data set and tested on the same separated dataset? Why the effort for using a different data set for each model?
I think you might be wrong here. Practitioners do train different models on the same data set and compare their ...
1
It seems that boosting algo is overfitting because train error is around 0 constantly. You could try to reduce cimplexity of the model e.g. try to use smaller tree depth or some other stopping criteria.
1
It is not only possible that different sets of predictors are used in different folds, it very likely in my experience.
It is however OK to average the error estimates from each fold. You want the instability of the predictor set to go into the error estimate: You are not estimating the expected error of a specific model, you are estimating the expected ...
1
differencing is a form of filtering . Unwarranted filtering like differencing or uneeded power transformations can have a deleterious effect.
the original series does not have to be stationary. the residuals from a useful model must be stationary.
1
The key idea is that cross-validation is not a method for finding out how well a model generalizes, but how well a procedure for fitting a model generalizes. So if your model has hyper-parameters that are tuned via cross-validation, then that is an integral part of the model fitting procedure and needs to be included in the outer cross-validation as well, ...
1
AFAIK, I have never seen approach #2 used, and although of course I guess you can always point to someone who have used it, it is certainly far from usual, a valid reason being what you have already mentioned, i.e. that you end up not using a portion of your data (the validation set) for fitting your final model. Save the fact that the phrase "CV on the ...
Only top voted, non community-wiki answers of a minimum length are eligible
