In model selection, what to do if expected prediction loss of all models is infinity? Consider choosing a model for prediction. The criterion is expected prediction loss: the lower the expected loss, the better the model. Suppose the distribution of prediction errors has relatively heavy tails for each of the models under consideration AND the loss function is such that the loss grows relatively fast w.r.t. prediction errors (e.g. quadratic loss or even exponential loss). Suppose that taken together, the distributions of prediction errors from different models and the shape of the loss function result in the expected loss being infinite. 
This makes it hard to choose among models as comparing infinity with infinity is hard. Note that both the distributions of prediction errors and the loss function are beyond out control (we have a given set of models and our client has specified the loss function reflecting his/her preferences). At the same time, the logic of choosing the model based on expected loss is appealing and we would probably like to stick to it (though if there is no feasible solution, we might be willing to give it up).
How do we choose a model in this situation?
 A: You actually have two "kinds" of expectation in your expected loss, at least in your current formulation:


*

*You take an expectation over the expected loss at a particular setting of the predictors

*You take an expectation of these expected losses over the distribution of predictors
(I'm not going to go into iterated expectations, Fubini etc. here, let's keep this hand-wavy.)
An infinite expected loss can occur at either one of these two points. You could have an infinite expected loss for a particular setting of your predictors. Or the expected loss could be finite for each particular predictor setting, but your predictors could simply be so far over the place that when you integrate over your predictors, you end up with infinite losses.
So I would say the first step should be to understand how losses at each particular predictor settings behave. Are there particularly problematic such settings, with infinite loss? If so, can we control the predictors so this situation does not occur? Once we have controlled the "pointwise" expected losses, can we constrain the possible predictor settings so that our many finite expected losses do not integrate up to infinity?
And if this doesn't work, i.e., you cannot even control the predictors to achieve "pointwise" finite expected losses, then I think you have a very strange problem on your hands, and you would need to go deeper into just what is happening here. I don't think this situation could be dealt with in general.
