Evaluating rare event risk metrics

Question

Suppose there is a rare event that happens on 3-7 days a year, and we are interested to predict days when it happens. We have two metrics, A and B, that both take values on onterval (0, 1) for any given day. Higher values of A on some day supposedly mean a greater chance of event happening on that day (but A is not a probability!), same goes for B.

We also have historical data for many years. For every day we know what was the value of A, what was the value of B and if the event really happened (denote as Y -- a binary variable). The question is: how to measure which metric, A or B, better predicts Y?

I have found a couple of useful ways:

• Mutual Information Score -- we could measure MI between A and Y, then between B and Y. The one with higher MI score wins;
• Average Precision Score -- roughly the area under precision-recall curve, the greater the area -- the better the metric.

However, I have found these two ways contradicting. Namely, in my case mutual information is greater for A, but the average precision score is greater for B. Why is this happening? What are some other metrics I could use, what are their pros and cons?

Answer 1

They are different because they measure completely different quantities. Mutual information is essentially the Kullback–Leibler divergence between the joint distribution and the product of the marginals, so it gives a measure of the total relationship between two variables. Average precision measures the strength of each metric when used as what is basically a logistic predictor with the slope fixed to one and the intercept set to the negative of the threshold. This follows from the latent variable model of logistic regression.

The benefits of MI over AP is that the latter only measures logistic relationships while the former looks at all relationships, including those that could result from some other non-linear transformation. So while B might be the better logistic predictor (I say might because AP only averages over a subset of logistic models), there is likely some non-linear function of A that is better overall. Of course, each MI is an estimate since you don't have the exact probability mass functions, and therefore its accuracy will depend on the number of samples provided.

With that in mind, if A and B are not highly correlated, you could try logistic regression on standardized A and/or B and compare regression coefficients, the fraction of deviance explained by each variable, likelihood ratios upon adding either variable to the model, Bayes factor, relative likelihood, etc. You could use non-linear terms and/or interaction terms, or use a general additive model to fit unknown smooth functions of A and/or B. However, this will really only help you to determine the form of the relationships as the MI is already a farely robust measure of their relative efficiency .