I am trying to get a deeper understanding of the common ML pipelines and I have some doubts regarding Cross Validation, why do we really use it and what does it really tell us about Bias and Variance.
Let's say I have a model, fit and tested. Accuracy is 82%. Sweet.
To my understanding, cross-validation now comes in saying "let's see if your results were due to sheer luck or you really rock it, let's repeat the test 5-10-N times". Therefore, CV improves the standard (single) train-test split and looks at "reproducibility". Either you keep this standard or your model is somehow "unstable". Here I would conclude that CV tests for the variance of the model, is it correct?
I mean, as far as I understood the decomposition between bias and variance is at the level of every single model, but since we cannot tell which part of the error is due to bias and which to variance, we can just take conclusions by comparing different results from different models. In other words, is it correct to say that with one single test we have no idea whatsoever about the variance of our model but thanks to CV we can make some guesses by comparing the performance of the N models from the N folds?
For example, here I have applied a 10-folds CV on my dataset and I have kept track of train and test accuracy (and cohen's kappa) for each fold. Then I plotted the results as follows:
What does this plot really tell us? I mean, I did CV to test for high variance, right? Does the volatility in the test accuracy give us a feeling for the variance of our model? Also, can we say something about bias?
And if I were right, how can I tell if the variance is actuall high? I have the feeling that I should compare these results with at least another model, for example trained with less features (because that would theoretically reduce the variance). In that case I would get the following:
Am I the only one that thinks I should quantify somehow some things in order to compare the models? So here I have more doubts:
- is it correct to compute for example the variance/standard deviation of the test accuracies as a comparable measure of variance? For example, I would compute the relative standard deviation (RSD) w.r.t. average test accuracy;
- is it correct to compute for example the average test accuracy as a comparable measure of bias?;
- how would you compare the two models to choose the best?.
In this case I would get the following results:
Metric | 1st Model | 2nd Model (Simpler) |
---|---|---|
Avg. Test Accuracy | 81.65% | 80.30% |
RSD Test Accuracy | 4.535% | 4.421% |
which somehow confirm that with simpler models the variance goes down at the price of higher bias. The thing is, does it make sense?