1
$\begingroup$

In a standard supervised learning setup, suppose I have a training set $X_{train}$, a validation set $X_{val}$ and a test set $X_{test}$. Assuming I have trained a final model $M$ on $X_{train} \bigcup X_{val}$, I now want to get an estimate of its generalization performance $P$. According to the literature, the generalization performance is estimated by scoring $X_{test}$ with $M$ and reporting the performance $\hat{P}$ .

Question 1: However, the above will give me a point estimate of the generalization performance. What are the standard methods to get a range estimate of generalization performance?

Question 2: If I had multiple test sets $X^{1}_{test}, X^{2}_{test}, X^{3}_{test}$... I suppose could estimate the average and standard deviation of performance across all test sets. However, as I only have one test set, would it be correct to bootstrap (sample with replacement) multiple test sets $X^{1'}_{test}, X^{2'}_{test}, X^{3'}_{test}$... and estimate average and standard deviation on those instead?

Note: Consider that collecting more data is not an option.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.