How do I
when I do repeated KFold cross validation using sklearn.model_selection.RepeatedKFold
I use the repeats as I have a small dataset (<200) and would like tighter bounds on my model performance for significance testing.
I am comparing Log Reg. vs Lasso vs Ridge vs GPs
I use the repeats as I have a small dataset (<200) and would like tighter bounds on my model performance for significance testing.
Repetitions of $k$-fold cross validation allow you to measure model instability, and the uncertainty related to that source of variance will go down if you average repetitions.
But it doesn't do anything about the actual number of independent cases in your data. So that part of the total variance uncertainty will not be improved by repeated cross validation. In fact, I don't think it can be improved by anything but larger sample size (and possibly a better model - but that you're measuring and thus you cannot directly influence it).
I usually have complex underlying data structure (≈ repeated measurements), so I bootstrap to get an idea of what is going on in this respect.
In addition, depending on the precise question you're tackling with your model, there may be further sources of variance which moreover cannot be measured with the cross validation experiment (e.g. in case you're working on algorithm comparison as opposed to building a model for production use from the data at hand).
So, if you can show that one of these sources of variance dominates your total uncertainty while the others are negligible you can construct your significance test based on that source of variance. Otherwise, you'll have to account for the different sources of variance.
