I think you may be misunderstanding conditional test error. This may be because Hastie, Friedman, and Tibshirani (HFT) are not consistent in their terminology, sometimes calling this same notion "test error", "generalization error", "prediction error on an independent test set", "true conditional error", or "actual test error".
Regardless of name, it's the average error that the model you fitted on a particular training set $\tau$ would incur when applied to examples drawn from the distribution of (X,Y) pairs. If you lose money each time the fitted model makes an error (or proportional to the error if you're talking about regression), it's the average amount of money you lose each time you use the classifier. Arguably, it's the most natural thing to care about for a model you've fitted to a particular training set.
Once that sinks in, the real question is why one should care about expected test error! (HFT also call this "expected prediction error".) After all, it's an average over all sorts of training sets that you're typically never going to get to use. (It appears, by the way, that HFT intend an average over training sets of a particular size in defining expected test error, but they don't ever say this explicitly.)
The reason is that expected test error is a more fundamental characteristic of a learning algorithm, since it averages over the vagaries of whether you got lucky or not with your particular training set.
As you mention, HFT show the CV estimates expected test error better than it estimates conditional test error. This is fortunate if you're comparing machine learning algorithms, but unfortunate if you want to know how well the particular model you fit to a particular training set is going to work.