2
$\begingroup$

I am dealing with a multi-class classification problem and I compute micro-averaged evaluation metrics (precision, recall and F-measure) by performing 10-fold cross validation. However, the fact that some classes have few data samples results in confusion matrices with no actual samples of these classes in the test set and no samples predicted as these classes. I present an example with 3 classes:

                                 Actual
                       | class 1 | class 2 | class 3 |
                            -----------
   Predicted | class 1 |    0    |    0    |    1    |
             | class 2 |    0    |    2    |    0    |
             | class 3 |    0    |    0    |    0    |

Class 1 has no actual data samples in test set and neither any data sample is predicted as of class 1. Class 2 has 2 True Positives, while Class 3 has one actual data sample that is predicted as of class 1. In that case, the micro-averaged precision is: $$Precision= \frac{\frac{0}{0} (class 1) + \frac{2}{2} (class 2)+ \frac{0}{1} (class 3)}{3}$$ By dividing with 0, I get Nan values in Matlab of course. My questions are:

  1. Is it correct to find mean by not taking into account the NaN part? In that case, micro-averaged precision in the example would be: $$Precision = \frac{\frac{2}{2} (class 2)+ \frac{0}{1} (class 3)}{2} = 0.5$$ But I think that this approach doesn't represent truly what happens with all classes.
  2. If the aforementioned approach is not correct, what can we do to overcome this problem? One thought would be to shuffle data samples before applying cross validation and apply cross-validation multiple times, until hopefully we get test sets with data samples from every class.

Thank you for your help.

PS: I have read related posts (e.g. this one) but I don't think they answer my question.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.