Is there a way to put confidence intervals around model evaluation metrics? This is more a theoretical question than something I'm working on now, but is a situation I see everywhere and would like some input into how to approach it. 
A modeler builds several competitor models on a training set and evaluates their performance on a holdout set by looking at some measure of fitness. Although one model seems to perform better by this measure, they both seem to be pretty close. What techniques are out there that can help us estimate if they are actually different for an arbitrary measure of fitness? 
My first idea was to take many bootstrap samples from the holdout set, calculate our fitness measure on each sample for both models, calculate the difference on each sample, and see where the bulk of this new distribution lies. If the bulk of the distribution is mostly on one side of zero, we might be able to make a determination regarding how often we expect one model to be better than another.
Are there problems with this approach? Are there better approaches? 
 A: Interesting question .. I have answered in a time series context because that is the only subject that I know anything about . I have implemented procedures that are used to estimate a family of forecast errors to enable model evaluation metrics to be generated.
After identifying a robust ARIMA model for a time series we have a set of model residuals. We can bootstrap say 1,000 forecasts for the next time period using monte carlo methods. An actual value for the next period is observed thus we now have 1,000 error measures for the robust ARIMA model that had been developed and consequentially upper and lower limits for that metric can be produced .
As a second candidate we could introduce an exogoneous/causal series and form a transfer function (regression on steroids !) and forecast the predictor and use those forecasts to aid the prediction of the endogneous series. Similar to approach 1 we can then form a distribution of forecast errors for the next period.
We can now generate and additional two sets of forecast errors by allowing for pulses to occur in the future thus in total four error distributions have been generated.
This process can be conducted for K periods into the future.
This would enable confidence limits to be placed around a forecast evaluation metric such as mape or mae . 
