2
$\begingroup$

I'm using a single classifier on two different datasets, consisting of non-overlapping set of observations. The performance on one dataset is higher than the other, and I want to show that the difference is statistically significant (or not). What test can I use for this? A reference to Python function would be great.

Please note that I'm not comparing two classifiers on a single dataset, but a single classifier on two datasets. Hence, the difference is on the separability of the datasets.

$\endgroup$
1
$\begingroup$

If you are interested in just the classification accuracy, you can do a $z$-test of the equality of two independent proportions. I don't know the Python code for that, but note that the test is equivalent to a chi-squared test, which should be easy to find / implement. You would have a 2x2 matrix of counts: the first row is classified correctly and the second row is incorrect, while the first column is for the first dataset and the second is for the second. Be aware that testing the classification accuracy constitutes a poor use of the information available to you; the test will have less power than a comparable test that does not use the predicted classes.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.