1
$\begingroup$

I have a classifier that is fixed and wish to evaluate its predictive performance using a test dataset. I'm familiar with the situation (e.g. in k-fold CV) where the data is split and the classifier trained on the subsets, but in this case I can't do any training. In particular I'd like confidence intervals on the F1 measure so am planning to bootstrap using the test dataset, calculating F1 with each bootstrap sample.

I can't find any info on this type of scenario - is the bootstrap a valid approach?

$\endgroup$
1
$\begingroup$

I am not an authoritative source, but it might help some thoughts.

I think that if the model is fixed than obviously k-fold CV does not make sense. It will produce the same error as if it were trained on the whole training data set.

Also because the model is fixed I think that it can be seen simply as a function of input. If the input is a random variable $X = X_1,X_2,..,X_n$, than our model is also a random variable $M = (M_1,M_2,..,M_n) = f(X) = f(X_1),f(X_2),..,f(X_n)$. We go further and say that F-measure $F_1$ is a function of some $M_i$ random variables. The point is that we can think of $F_1$ as a random variable which has an unknown distribution, but based on the transformations through the classifier we can obtain a sample for this $F_1$ random variable. Thus percentile bootstrapping looks to me a valid way to get confidence intervals for this random variable based on the sample $m = f(x) = f(x_1),f(x_2),..,f(x_n).$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.