# Methods to produce range estimates on test set

In a standard supervised learning setup, suppose I have a training set $$X_{train}$$, a validation set $$X_{val}$$ and a test set $$X_{test}$$. Assuming I have trained a final model $$M$$ on $$X_{train} \bigcup X_{val}$$, I now want to get an estimate of its generalization performance $$P$$. According to the literature, the generalization performance is estimated by scoring $$X_{test}$$ with $$M$$ and reporting the performance $$\hat{P}$$ .

Question 1: However, the above will give me a point estimate of the generalization performance. What are the standard methods to get a range estimate of generalization performance?

Question 2: If I had multiple test sets $$X^{1}_{test}, X^{2}_{test}, X^{3}_{test}$$... I suppose could estimate the average and standard deviation of performance across all test sets. However, as I only have one test set, would it be correct to bootstrap (sample with replacement) multiple test sets $$X^{1'}_{test}, X^{2'}_{test}, X^{3'}_{test}$$... and estimate average and standard deviation on those instead?

Note: Consider that collecting more data is not an option.