The meaning of conditional test error vs. expected test error in cross-validation

Question

My textbook on cross-validation is The Elements of Statistical Learning by Hastie et al. (2nd ed.). In sections 7.10.1 and 7.12, they talk about the difference between conditional test error $$E_{(X^*,Y^*)}[L(Y, \hat{f}(X))|\tau]$$ and expected test error $$E_\tau [E_{(X^*,Y^*)}[L(Y, \hat{f}(X))|\tau]].$$ Here $\tau$ is the training data set, $L$ is the loss function, $\hat{f}$ is the model trained on $\tau$. $E$ is the expectation.

They explained that CV only estimates the expected test error well.

My question is, is there any reason why we would care about the conditional test error?

The only reason I could think of is that we want to answer the question 'If God puts $n$ data sets on the table, but only lets us take 1 home to fit our model, which one should we choose?'

Answer 1

I think you may be misunderstanding conditional test error. This may be because Hastie, Friedman, and Tibshirani (HFT) are not consistent in their terminology, sometimes calling this same notion "test error", "generalization error", "prediction error on an independent test set", "true conditional error", or "actual test error".

Regardless of name, it's the average error that the model you fitted on a particular training set $\tau$ would incur when applied to examples drawn from the distribution of (X,Y) pairs. If you lose money each time the fitted model makes an error (or proportional to the error if you're talking about regression), it's the average amount of money you lose each time you use the classifier. Arguably, it's the most natural thing to care about for a model you've fitted to a particular training set.

Once that sinks in, the real question is why one should care about expected test error! (HFT also call this "expected prediction error".) After all, it's an average over all sorts of training sets that you're typically never going to get to use. (It appears, by the way, that HFT intend an average over training sets of a particular size in defining expected test error, but they don't ever say this explicitly.)

The reason is that expected test error is a more fundamental characteristic of a learning algorithm, since it averages over the vagaries of whether you got lucky or not with your particular training set.

As you mention, HFT show the CV estimates expected test error better than it estimates conditional test error. This is fortunate if you're comparing machine learning algorithms, but unfortunate if you want to know how well the particular model you fit to a particular training set is going to work.

Answer 2

I'm thinking about the same passage and am also wondering when I would ever be interested in the conditional test error. What's more, as far as I can understand they should be the same asymptotically: for very large training and test sets the precise training/test set split should no longer result in different conditional test error estimates. As you can see in the Hastie et al. book their examples on conditional - expected differences are always based on relatively small number of observations, which if I understand this correctly is the reason for why conditional and expected test errors look different in the graphs.

The book mentions that the expected test error averages over the randomness in the training set, while the (conditional) test error does not. Now when would I want to take the uncertainty associated with which particular training/test-set partition I make into account? My answer would be that I'm usually never interested in accomodating this kind of uncertainty as this is not what I'm interested in when I'm doing model assessment: In assessing a the predictive quality of a model I want to know how it would fare in let's say forecasting the weather tomorrow. The weather tomorrow is related to my overall data pretty much as my test data is related to my training data - so I calculate one conditional test error to assess my model. However, the weather tomorrow is related to my overall data not like one specific test set is related to the corresponding specific training set, but how the average test set is related to the average training set. So I obtain the next training/test- set partition and get another conditional test error. I do this many times (as e.g. in K-fold cross-validation) - the variation in the individual conditional test errors averages out - and I'm left with the expected test error; which, again, is all I can think of wanting to obtain.

Put differently, in the test error/expected test error graphs in Hastie et al., we get an idea of the efficiency of the model estimator: if the conditional test errors are widely dispersed around the expected test error this is an indication of the estimator being inefficient, while less variation in the conditional test errors would indicate a more efficient estimator, given the amount of observations.

Bottomline: I might be mistaken here, and I'd be happy to be corrected on this, but as I see it at the moment the concept of the conditional test error is a doubtful attempt at assessing external model validity through allowing oneself only one training/test-partitioning shot. For large samples this single shot should be equivalent to conditoinal test errors averaged over many training/test-partitioning shots, i.e. the expected test error. For small samples where a difference occurs the actual measure of interest appears to me to be the expected, and not the conditional test error.