In a standard supervised learning setup, suppose I have a training set $X_{train}$, a validation set $X_{val}$ and a test set $X_{test}$. Assuming I have trained a final model $M$ on $X_{train} \bigcup X_{val}$, I now want to get an estimate of its generalization performance $P$. According to the literature, the generalization performance is estimated by scoring $X_{test}$ with $M$ and reporting the performance $\hat{P}$ .

Question 1: However, the above will give me a point estimate of the generalization performance. What are the standard methods to get a range estimate of generalization performance?

Question 2: If I had multiple test sets $X^{1}_{test}, X^{2}_{test}, X^{3}_{test}$... I suppose could estimate the average and standard deviation of performance across all test sets. However, as I only have one test set, would it be correct to bootstrap (sample with replacement) multiple test sets $X^{1'}_{test}, X^{2'}_{test}, X^{3'}_{test}$... and estimate average and standard deviation on those instead?

Note: Consider that collecting more data is not an option.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.