What do Cross Validation results actually tell about Bias and Variance?

Question

I am trying to get a deeper understanding of the common ML pipelines and I have some doubts regarding Cross Validation, why do we really use it and what does it really tell us about Bias and Variance.

Let's say I have a model, fit and tested. Accuracy is 82%. Sweet.

To my understanding, cross-validation now comes in saying "let's see if your results were due to sheer luck or you really rock it, let's repeat the test 5-10-N times". Therefore, CV improves the standard (single) train-test split and looks at "reproducibility". Either you keep this standard or your model is somehow "unstable". Here I would conclude that CV tests for the variance of the model, is it correct?

I mean, as far as I understood the decomposition between bias and variance is at the level of every single model, but since we cannot tell which part of the error is due to bias and which to variance, we can just take conclusions by comparing different results from different models. In other words, is it correct to say that with one single test we have no idea whatsoever about the variance of our model but thanks to CV we can make some guesses by comparing the performance of the N models from the N folds?

For example, here I have applied a 10-folds CV on my dataset and I have kept track of train and test accuracy (and cohen's kappa) for each fold. Then I plotted the results as follows:

What does this plot really tell us? I mean, I did CV to test for high variance, right? Does the volatility in the test accuracy give us a feeling for the variance of our model? Also, can we say something about bias?

And if I were right, how can I tell if the variance is actuall high? I have the feeling that I should compare these results with at least another model, for example trained with less features (because that would theoretically reduce the variance). In that case I would get the following:

Am I the only one that thinks I should quantify somehow some things in order to compare the models? So here I have more doubts:

• is it correct to compute for example the variance/standard deviation of the test accuracies as a comparable measure of variance? For example, I would compute the relative standard deviation (RSD) w.r.t. average test accuracy;
• is it correct to compute for example the average test accuracy as a comparable measure of bias?;
• how would you compare the two models to choose the best?.

In this case I would get the following results:

Metric	1st Model	2nd Model (Simpler)
Avg. Test Accuracy	81.65%	80.30%
RSD Test Accuracy	4.535%	4.421%

which somehow confirm that with simpler models the variance goes down at the price of higher bias. The thing is, does it make sense?

Answer 1

This is just a continuation of my comments above. The short answer to

Why do we really use cross-validation (CV)?

is to get an estimate of the risk of an estimator. This could be the MSE in regression, misclassification rate (= 1 - accuracy) in classification, and so on.

Splitting into the training and test sets, once, and computing the accuracy over the test set is a form of CV. So when you say

... fit and tested. Accuracy is 82%.

You have done a form of CV to get this estimate 82%. This would be a very variable estimate if you compute it based on a single random split. If you split the data at random a second time, you get a different result. Now if you average these two values (a form of 2-fold CV), this would be a better estimate of the accuracy, i.e., will have a lower variance.

Alternatively, you can use the standard 2-fold CV: Split into equal-sized portions (at random), train on one and test on the other and vice versa; then take the average of the two values. All this does is give you a less variable estimate of the accuracy than your original single-split approach.

---

Part of the standard K-fold cross-validation procedure is shuffling the data at random. For each random permutation you get a different result. This variation tells you something about the variance of the estimate you obtain for the score/risk/etc.

Alternatively, you can split into training and test K times at random, and take the average over K splits. The result would be very close to K-fold cross-validation. You can do this procedure multiple times and you get an estimate of the accuracy and a variance for your estimate of the accuracy.

Here is some code:

import numpy as np 
from sklearn.model_selection import train_test_split
from sklearn import datasets
from sklearn import svm
from sklearn.model_selection import cross_val_score
import matplotlib.pyplot as plt

X, y = datasets.load_breast_cancer(return_X_y=True)
n, _ = X.shape

# flip some labels to make it harder
idx = np.random.choice(n, int(np.floor(0.2*n)), replace=False)
y[idx] = np.mod(y[idx]+1,2) 

# Pick only the first two features
Xs = X[:,[0,1]] 

nperms = 100
nfolds = [1, 2, 5, 10]
res = np.zeros((len(nfolds), nperms))

for fi, nfold in enumerate(nfolds):
    print(nfold, end=' ')
    for t in range(nperms):
        dat = np.hstack((Xs, y.reshape((-1,1))))
        dat = np.random.permutation(dat)
        yp = dat[:,2]
        Xp = dat[:,0:2]

        clf = svm.SVC(kernel='linear', C=1)
        if (nfold == 1):
            X_train, X_test, y_train, y_test = train_test_split(Xp, yp, test_size=0.5)
            clff = clf.fit(X_train, y_train)
            res[fi, t] = clff.score(X_test, y_test)
        else: 
            scores = cross_val_score(clf, Xp, yp, cv=nfold)
            res[fi, t] = scores.mean()

plt.scatter(np.array(nfolds).repeat(nperms), res.flatten())
plt.savefig('cv_test.png')

res.std(axis=1)

This is what the result looks like:

The standard deviations for the estimate of the accuracy, from the last line of code, are:

array([0.0209317 , 0.00511349, 0.00426156, 0.00382858])

---

Since your task is classification, you are looking at the 0-1 loss. The resulting risk (= 1 - accuracy) depends on more than just the bias and variance of the estimator, i.e., knowing these is not enough to compute the accuracy.

(The term "variance of the model" in the question is not clear. We have the variance of an estimator. A model can produce multiple estimators: an estimator of the parameters of the model, an estimator of the response, an estimator of the risk of the model, etc.)

In regression with a quadratic risk, the story is different. This particular risk has the bias-variance decomposition.

Also, it worth mentioning that you have two things here:

1. An estimator of the parameters of the model.
2. An estimator of the risk of the estimator (1).

CV is a way to construct (2) and this estimator of the risk itself has variation as the above code demonstrates (sort of). You can lower this variation to some extend by using more folds (and you can see that there is a limit).