Suppose there is a rare event that happens on 3-7 days a year, and we are interested to predict days when it happens. We have two metrics, A and B, that both take values on onterval (0, 1) for any given day. Higher values of A on some day supposedly mean a greater chance of event happening on that day (but A is not a probability!), same goes for B.
We also have historical data for many years. For every day we know what was the value of A, what was the value of B and if the event really happened (denote as Y -- a binary variable). The question is: how to measure which metric, A or B, better predicts Y?
I have found a couple of useful ways:
- Mutual Information Score -- we could measure MI between A and Y, then between B and Y. The one with higher MI score wins;
- Average Precision Score -- roughly the area under precision-recall curve, the greater the area -- the better the metric.
However, I have found these two ways contradicting. Namely, in my case mutual information is greater for A, but the average precision score is greater for B. Why is this happening? What are some other metrics I could use, what are their pros and cons?