# Performance metric for algorithm predicting probability of low probability events

• click through rates are very small (like 0.1%)

• click through rates depend on several parameters (like size of the ad, country in which ad is shown, whether the user has seen this ad earlier etc)

As I employ statistical/machine learning techniques, I would like to measure the performance of my techniques on historical data. I cannot employ metrics like precision or accuracy since all I predict is probability rather than predicting occurrence or non-occurrence of event.

In such a case, how should I measure the performance of my algorithm?

• You say that you want to predict the CTR instead of whether a single add has been clicked or not. How does the feature "whether the user has seen this ad earlier" fits then ? I'd assume that one observation is add + ctr, not add + click yes/no. May 21, 2012 at 8:35
• Hi steffen, "whether the user has seen this ad earlier" has an effect on ctr: ctr is higher when the ad being shown to the user for the first few times (because of novelty), and lower subsequently. I should have rather put the metric as "number of times user has seen this ad earlier". May 22, 2012 at 4:57

Mean squared error as suggested by Lakret will certainly work, however, I'd like to propose a method which captures the uncertainty of the clickrates of the adds (which are not known, exactly, but only estimated from historic data).

Let's say we have an add in our validation set with 10000 showns and 10 clicks, i.e. the maximum likelihood estimate for the clickrate $p$ is $0.001$. Furthermore we predicted a clickrate of $\hat{p}$ for this add.

Now instead of comparing the predicted $\hat{p}$ with p, we check whether $\hat{p}$ is in the confidence interval of $p$. Using the Beta-Distribution aka the Bayesian approach to calculate the confidence interval (called credible intervals then), we get using R

alpha <- 0.05
qbeta(c(alpha/2,1-alpha/2),10+1,10000-10+1)
# which results in  0.000549185 0.001838080


For other methods to calculate binomial confidence intervals see e.g. the R-package confint.

Now, the error for the prediction of a single add is ...

• 0, if $\hat{p}$ is in the confidence interval of p
• 1, else

Starting from here, one can calculate binomial metrics like precision OR just the average error across multiple clickrate predictions. In a more sophisticated approach one could calculate the error as the distance to the nearest confidence interval bound (if outside the confidence interval) to make the error less discrete.

Since your actual data is bivariate (click/no click) you should make your predictions discrete as well (e.g. if predicted probability is more than some value then assign it 1). Then you can employ metrics like precision or accuracy.

• That might have worked, but on some thinking I found a (hopefully) better idea. May 22, 2012 at 4:50
• Accuracy seems like a pretty poor metric here. If click through rate is .1%, I can come up with a model that just says 'no click through' every time and be 99.9% accurate. May 7, 2018 at 16:31

Mean absolute error and Mean squared error may be helpful.

why not simply use the correlation coefficient between the predicted click probability and the click event (0 or 1)? Higher the correlation, better the algorithm.

• Even if the predicted click probability is exactly equal to the true parameter $p$, the Pearson correlation coefficient $r$ will not necessarily equal 1, so, while this method does have some merits, it can be hard to interpret / prone to misinterpretations. Oct 5, 2012 at 14:15

I found this approach works well:

If there are $N$ examples where we have to calculate CTRs, following is a metric for the performance of algorithm:

Let our algorithm predict a $ctr$ of $ctr(i)$ for example $i$: $$v = \sum_{i =1}^{N}\frac{p(i)}{N}$$ where

$p(i) = ctr(i)$ if actually there was a click

$p(i) = 1 - ctr(i)$ if actually there was not a click

Now, $v$ will range from 0 to 1, and will be higher for better prediction algorithms.

• You're making your life hard. You're doing exactly the same as what @danas.zoukas suggested, except you're averaging across multiple examples. First, you're depriving yourself of all the analytical tools that come with metrics such as precision. Second, the probability doesn't really exist - the observations do. May 22, 2012 at 5:35