I would compare two classifiers (A and B), where B is obtained from A.
I would exploit the F-measure computed on two samples (of two independent populations, p1 and p2, respectively):
A -> F-measure1 on p1
B -> F-measure2 on p2
Once obtained an F-measure index for the first and the second classifier, I might execute an hypothesis testing in order to measure if index of second classifier (F-measure2) is greater or smaller and such difference is statistically significant.
Does it make sense? Can I use the typical hypothesis testing for average of a population, substituting average with F-measure?