0
$\begingroup$

I have a Bayesian model that predicts the chance of some event occurring. This is basically a binary classification. To view the accuracy, I took the prediction output of my test data and binned it every 10%.

For example, if the model predicted a 15% chance of the event occurring I placed it in the 10% - 20% bin.

Once I had binned all the test data, I then calculated the actual probability for that bin. Using the last / far right bin as an example, the predicted was 90-100% while the actual was 97.2%. For this model, the prediction and the actual are fairly close.

Accurate Model accurate model

In this model (below), the predicted and the actual are off for most of the bins. In other words, my model isn't predicting the event very well. My question is, how do I describe / quantify the error between what the model says will happen vs what really happens.

I was thinking something like, the probability of this event happening is 22% +- 6%, but I have no idea how to calculate that.

Inaccurate mdoel inaccurate model

$\endgroup$

1 Answer 1

1
$\begingroup$

It might be better to plot the differences rather than describe them in words.

Suppose you have $n$ observations, $D_1,\ldots,D_n$, with associated model probabilities $p_1,\ldots,p_n$. The number of times you expected to see datum $D_i$ is distributed as $\textrm{Binomial}(n,p_i)$, from which you can calculate the mean $m_i$ and, say, 10%/90% quantiles, $u_i$ and $v_i$. In other words, you expect your $D_i$ to be scattered around the predicted $m_i$ and, 80% of the time, $u_i\le D_i\le v_i$.

OK, so then on the basis that a picture is worth a thousand words, you could represent your model's performance with a simple scatterplot. I've shown one below for data that does fit the model. The scatterplot may be hard to read in the bottom left where the counts are small, so you could also consider shifting and rescaling one of the axes (I subtracted $m_i$ then divided by $\sqrt{m_i}$) for ease of viewing: that's the second plot. enter image description here

Be aware that it's easy to be misled by plots like these, as the eye is naturally drawn to outliers (the bottom right of the rescaled plot looks a little dicey, doesn't it?). It's always worth generating some data from the model and plotting those observed vs expected, too, just to get an intuitive feel for what "ordinary" variation from the mean looks like.

$\endgroup$
5
  • $\begingroup$ Thanks for the thoughts. I may be missing something so please bear with me. By binning my predicted values haven't I essentially already done what you're suggesting in the first paragraph? The graphs themselves have an observed axis. Given that the event is binary (it either happened or it didn't) there's no way to calculate an observed probability for each observation. I would need to bin them them to calculate an average which is what I did originally. $\endgroup$
    – Preston
    Commented Sep 8, 2015 at 15:42
  • $\begingroup$ Hi Preston. I'm probably misunderstanding your set-up; tell me where I'm going wrong. You are observing events, $E$. Your events can take values $e_1, e_2, \ldots, e_n$ and you have a model that says how likely each of those $e_i$ is. OK, what's $n$ (the number of possible outcomes), and how many events do you actually observe? Finally, what hoping for when "quantifying the error" -- to identify the parts of the model that are wrong? To improve the model? To summarise the performance of your model for the reader of a report? Sorry ... so many questions! $\endgroup$
    – Creosote
    Commented Sep 8, 2015 at 19:39
  • $\begingroup$ This is a binary classification, so one event with two outcomes. The model is producing the probability that the event will happen for any given observation. I then bin the output prediction with similar output predictions. Depending on the bin, that probability appears to be good because the actual probability of the event happening is similar to the predicted. Where I'm concerned are the bins where the predicted is considerably different than the actual(the second graph I posted has several of these). It's this difference that I want to quantify. Does that help? $\endgroup$
    – Preston
    Commented Sep 8, 2015 at 23:46
  • 1
    $\begingroup$ Good solution. In addition, I would try scatterplots using filled points with some transparency. This will help with potential overplotting, and help avoid the problem of drawing the eye toward outliers. $\endgroup$ Commented Sep 8, 2015 at 23:48
  • $\begingroup$ Is this a Naive Bayes model, so $\mathbb{P}(E|x_1,\ldots,x_r) \propto q(E)\times p_1(E|x_1)\ldots p_r(E|x_r)$ ? If so, you should probably bin your data not by $\mathbb{P}(E|{\bf x})$ but by the value of some of the $x_k$ variables, e.g. by $x_3$ = "apple" or "banana" or "custart tart" ... (or whatever); then you can report the accuracy of the model on those slices of the data. You should also explore the naive independence assumption by calculating correlation coefficients for pairs of explanatory variables, $(x_j,x_k)$, to understand how the model is breaking down. $\endgroup$
    – Creosote
    Commented Sep 9, 2015 at 6:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.