5
$\begingroup$

Suppose I have a bag of marbles containing blue and red marbles. I guess/predict that my chance of drawing a red marble is 60%.

If the actual distribution turns out to be 80% blue/20% red, how can I best quantify the accuracy of my initial prediction?

Is it possible to restrict such a quantification to a range between 0 and 100%, such that a correct prediction (20%) would evaluate to a 100% accuracy?

$\endgroup$

2 Answers 2

3
$\begingroup$

I'll take a somewhat different track from Demetri. There are multiple aspects here.

A best quantification of prediction accuracy is surprisingly tricky. An accuracy measure is, for practical purposes, a mapping $f$ that takes your point prediction $\hat{p}$ (the predicted proportion of blue marbles, say) and a random sample $x$ from the bag and returns an "accuracy" value. "For practical purposes", because this again relies on sampling from the bag, because we will usually not know the true proportions. This is Demetri's argument.

So we now are faced with an $f(\hat{p},x)$ that is a random variable, because it depends on the sample $x$. (Typically, we would sample multiple $x$ and take the average, $\frac{1}{n}\sum_{i=1}^n f(\hat{p},x_i)$.) We can now take a look at the statistical properties of our random variable $f(\hat{p},x)$ as a function of $\hat{p}$. For instance, we can require that the expected accuracy be maximized of the predicted proportion $\hat{p}$ is equal to the true proportion $p$ in expectation. This will quickly lead to proper scoring rules and the log loss (another term for the entropy Demetri notes). The log loss has this property: it will elicit an unbiased estimate $\hat{p}$ of $p$.

Alternatively, we could of course look at error measures we want to minimize (instead of maximizing accuracy - it's the same thing, of course).

Here is a trap for the unwary. You might be tempted to use something like the Mean Absolute Error between $\hat{p}$ and the proportion of blue marbles you draw. This amounts to $$ f(\hat{p},x)=\begin{cases} 1-\hat{p} & \text{if $x$ is blue} \\ \hat{p} & \text{if $x$ is red} \end{cases} $$ This looks similar but simpler than the log loss. Instead of the log, we essentially have an absolute value. However, here is the problem: the MAE will not be minimized by the true proportion. Rather, if $p>0.5$, the MAE will be minimized by setting $\hat{p}=1$, regardless of the actual value of $p$!

You can either calculate the expected value of the MAE for each given $\hat{p}$ analytically, or trust a little simulation: for a true $p=0.8$, I looked at $\hat{p}=0, 0.01, 0.02, \dots, 1$, simulated $10,000$ draws from the urn in each case, calculated the MAE, and plotted the MAE against the prediction $\hat{p}$. While I was at it, I did the same using the Mean Squared Error (AKA the Brier Score) and the log loss instead of the MAE (with a minus sign, so we want to minimize the log loss), and added a red vertical line at the true proportion $p=0.8$:

simulated errors

As you see, using the MAE will reward us more (have lower errors) for a biased prediction $\hat{p}=1$ than for the unbiased estimate $\hat{p}=p=0.8$. (If $p<0.5$, the MAE will like $\hat{p}=0$ best.) Thus, if you want an unbiased estimate of $p$, you should not use the MAE. Alternatives include the Mean Squared Error, or the log loss per Demetri.

Well, now we have two error measures, the MSE and the log loss, that will at least not systematically mislead us (like the MAE). How do we choose between these two? (And of course, there are infinitely many other such error metrics which will elicit an unbiased prediction.) Now the question is how painful deviations from the true $p$ are for you. As you see in the simulated plots, as the predicted proportion $\hat{p}$ varies away from the true $p$, the log loss and the MSE behave rather differently. And you can of course use transformations of these error metrics to change their behavior further. As Demetri writes, it comes down to whether a prediction of $\hat{p}=0.6$ or $\hat{p}=1$ is more painful if the true $p=0.8$ - where in practice you will not know the true $p$. A few thoughts specifically on the comparison between the log loss and the MSE/Brier score can be found at Why is LogLoss preferred over other proper scoring rules?

Finally, one last aspect. You ask about "standardizing" error metrics to the interval $[0,100\%]$. As Demetri writes, I do not think this is truly possible. On the one hand, you would need to know when you have reached 100% accuracy, and in a practical situation, this simply won't happen. Similarly, it's hard to say whether you have reached 99% or 99.9% accuracy. It might be easier to assign 0% accuracy to a prediction - if you are certain that $p>0.5$, then you could say that a prediction of $\hat{p}=0$ is "as wrong as it can be", and you could say that this prediction is 0% accurate. Same for $\hat{p}=1$ if you know that $p<0.5$. The problem here is if you are unsure about the true $p$. And of course this only works because we know (actually do know here!) that $0\leq p\leq 1$ - if you are predicting, say, temperatures, it's very hard to say what is the "worst possible prediction" that we would like to say is 0% accurate.

R code for the plots:

true_prop <- 0.8
predictions <- seq(0,1,by=.01)
sims <- runif(10000)<true_prop

maes <- sapply(predictions,function(xx)mean(abs(xx-sims)))
mses <- sapply(predictions,function(xx)mean((xx-sims)^2))
logs <- -sapply(predictions[-c(1,length(predictions))],
    function(xx)mean(c(rep(log(xx),sum(sims)),rep(log(1-xx),sum(!sims)))))

par(mfrow=c(1,3),mai=c(.8,.5,.5,.1),las=1)
plot(predictions,maes,type="l",xlab="Predicted proportion",
    ylab="",main="Mean Absolute Error")
abline(v=true_prop,col="red")
plot(predictions,mses,type="l",xlab="Predicted proportion",
    ylab="",main="Mean Squared Error/Brier Score",las=1)
abline(v=true_prop,col="red")
plot(predictions[-c(1,length(predictions))],logs,type="l",
    xlab="Predicted proportion",ylab="",main="Log Loss",las=1)
abline(v=true_prop,col="red")
$\endgroup$
1
  • 1
    $\begingroup$ Thanks for the helpful response (as always) and providing the R code for the helpful plots. I've created a reproducible Python version here in case anyone is looking to replicate this in Python. $\endgroup$ Jun 28 at 17:00
3
$\begingroup$

Is it possible to restrict such a quantification to a range between 0 and 100%, such that a correct prediction (20%) would evaluate to a 100% accuracy?

I don't believe so. What you're asking is "is there a way to know when my prediction is equal to the truth" and that just can't be done. That is the entire reason we need statistics.

There are ways to estimate what the prediction should be after having seen data. These methods find the prediction which would minimize an appropriate error metric, and so the prediction would be the best prediction you could make conditioned on the data you had seen so far.

As an example, suppose you pull 10 marbles, 3 of which are red. One way to measure your error is through the cross entropy loss.

$$ \ell(\theta) = - \sum_i y_i \ln(\theta) + (1-y_i)\ln(1-\theta) $$

Here, $\theta$ is your prediction, $y_i=1$ is the event you get a red marble, and $y_i=0$ is the event you get a blue marble. Smaller numbers are preferable here. Using your 60% initial prediction your error would be approximately 8. Is that number bad? Is that number good? Tough to say, we need something to contrast against.

Let's use that data to create a new prediction. We saw that 30% of the marbles were red, so let's devise a new prediction: 30% of the samples are red.

We pull a new sample: 2 reds 8 blues. Now, let's compare the loss values of our initial prediction of 60% with our new prediction of 30%.

Error for 60% prediction: 8.345

Error for 30% prediction: 5.26

Our new prediction is much much better because it resulted in smaller error(in fact, it is the best prediction we could make having only seen those first 10 draws).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.