3
$\begingroup$

I have a logistic regression recommender model built on my data where I tried to predict one of two outcomes for each row. Let's call them success and fail. I'm using the cross_val_score function of SKLearn to do so. I was planning on just using precision and recall as performance measures, but saw the "micro" and "macro" options and decided to read up more and try those as well, but I'm confused by the results. For reference, I looked at this blog post, this SO question, and this paper.

While I think I understand the general concept, I'm confused by the results I'm getting, and which one is ideal for my use case. One important piece of background is that I have fewer success cases that fail. success makes up 25% of my dataset.

My understanding is that micro is a value that's closer to the performance of the model on the larger class (in this case, fail), while macro is for the smaller one (success). This makes sense. I'm trying to predict successes in this case, so the latter should be the better choice.

Here are the parts I don't get:

a) How do each of these compare to just regular precision / recall, in terms of representing the larger or smaller classes? Which of the three options is optimal for a case like this where I want to surface the smaller class but not the large one.

b) In some folds of my CV, I get 0 as the regular precision value, but none are 0 for micro and macro. How is that possible? If I get some true positives in the individual classes, shouldn't I always have some in the overall dataset as well? In general the regular precision and recall end up significantly lower than both types of averages.

Improve this question
$\endgroup$
1
  • $\begingroup$ Are you building a binary classifier (i.e. only two output classes)? Because then you can just use the regular precision and recall values. I like to create a visualization like this for binary classifiers: scikit-yb.org/en/latest/api/classifier/threshold.html $\endgroup$
    – Dan
    Commented Oct 26, 2018 at 10:21

1 Answer 1

Reset to default
1
$\begingroup$

These micro and macro averaging techniques are typically made for situations with more than 2 classes. With two classes you would only compute precision, recall, F1 or whatever you are interested in (you tell us! We cannot know) with regard to almost always your minority class (since it almost always is the one you're interested in).

If these averaging techniques use arithmetic means (on the micro or macro level), the other non 0 value will make the average larger than 0 as in $(0 + 5)/2=2.5$. If they used geometric averages (which is very uncommon), then $\sqrt{0\times5}=0$

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.