1
$\begingroup$

I am estimating the probability for an event to happen based on certain criteria prior to the event. The event either happens or does not happen (1 or 0). I have a few hundred thousand such pairs of 1) prior estimation in % and 2) 1/0 (happens/does not happen).

My question is: What is the best scientific method to test how good my estimations were? My hypothesis is that I am over/underestimating the probability "in the tails" (e.g. near to 0% or near to 100%). I have tried splitting the pairs into a number of ranges, which I have arbitrarily determined (e.g. 0-10%, 10%-20%, ... etc) to see how close I am in each bracket, but the results do not seem very revealing. I would be thankful if someone told me a better way or at least give me some direction.

Improve this question
$\endgroup$
2
  • $\begingroup$ It is not clear how you get this histogram of prior percentages. How do you get the pairs? $\endgroup$
    – Michael R. Chernick
    Commented May 11, 2017 at 20:50
  • $\begingroup$ I am using sports betting odds as estimate for the probability of an event happening. However they deduct certain profit margin from the "real" odds they estimate. I am trying to understand how exactly do they do that. So I am using different ways of adding back the margin to the market odds to arrive at the real odds and try to see which model works best. $\endgroup$
    – Nenko
    Commented May 11, 2017 at 21:03

1 Answer 1

Reset to default
2
$\begingroup$

Density or probabilistic forecasts can be evaluated using so-called (you may want to look through previous questions with this tag). A scoring rule is simply a function that maps a probabilistic forecast ("60% chance of the event happening, 40% of not") and the corresponding outcome to a value. These scores are then averaged. Crucially, scoring rules can be used for probabilistic forecasts that vary between events that you forecast. The disadvantage is that scoring rules are not overly intuitive, and you can't say what a "good" value is.

An early paper for dichotomous forecasts is Murphy & Winkler, 1987, Monthly Weather Review. More recently, Tilman Gneiting has been working extensively on this. One review is Gneiting & Katzfuss, 2014, Annual Review of Statistics and its Application. Gneiting & Raftery (2007, JASA) give examples for (strictly) proper scoring rules for discrete variables. If you are most interested in specific values of your prediction, then Gneiting & Ranjan, 2011, Journal of Business & Economic Statistics or Diks et al., 2011, Journal of Econometrics may be helpful.

Let's look at an example. Suppose we have a dichotomous variable. We have three different potential predictive densities: our predictive density is one of

  • either $\widehat{p_0}=1$ and $\widehat{p_1}=0$
  • or $\widehat{p_0}=0.5$ and $\widehat{p_1}=0.5$
  • or $\widehat{p_0}=0$ and $\widehat{p_1}=1$

Suppose further that the true future density is also one of these. We thus have $3\times 3=9$ potential combinations of predictive and true densities.

We assess predictive densities using the Brier or quadratic score. For an outcome $i$, this is

$$ -2\widehat{p_i} + \sum_{j=0}^1 \widehat{p_j}^2,$$

of which we will look at the expectation. The first two cases are:

  1. The predictive distribution is $\widehat{p_0}=1$ and $\widehat{p_1}=0$, and this is indeed the true distribution. Then the expected Brier score is $$ -2(1\times1 + 0\times0)+1 = -1.$$
  2. The predictive distribution is $\widehat{p_0}=\widehat{p_1}=0.5$, but the true distribution is $p_0=1$ and $p_1=0.5$. Then the expected Brier score is $$ -2(1\times 0.5+0\times 0.5)+0.25+0.25 = -0.5.$$

The full table is: $$ \begin{array}{c|ccc} & p_0=1, p_1=0 & p_0=p_1=0.5 & p_0=0, p_1=1 \\ \hline \widehat{p_0}=1, \widehat{p_1}=0 & -1 & 1 & 1 \\ \widehat{p_0}=\widehat{p_1}=0.5 & -0.5 & -0.75 & -0.5 \\ \widehat{p_0}=0, \widehat{p_1}=1 & 1 & 0 & -1 \end{array} $$

We see how in each column, the correct predictive distribution gives the lowest expected Brier score, in the diagonal cell. This is not overly surprising, since by Gneiting & Raftery (2007, JASA), the Brier score is strictly proper, i.e., it is minimized in expectation by the correct future distribution, and by no other.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I have read into the different scoring rules and so far I don't really "like" any of them. For example Brier and logaritmic score rules. If I assign 50% probability to 100 events in a row, whether I get a.100 0's, b. 100 1's or c. 50 0's and 50 1's, all 3 options will give me the same score, while obviously I made good predictions in option c. and terrible ones in a. and b. So scores for predictions of 50% and the surrounding region seem useless. Do you have any idea of a scoring rule that addresses this issue? $\endgroup$
    – Nenko
    Commented May 11, 2017 at 22:39
  • $\begingroup$ Sorry for taking a while. I think your last comment is looking at things the wrong way: scoring rules don't compare a single predictive distribution across multiple future true distributions (rows in the table I added) - rather, they allow you to compare different predictive distriibutions for a single true future distribution (columns in the table). After all, that's what we are interested in: to find the best predictive distribution for a future series of outcomes, not to find one predictive distribution that fits widely different future distributions. Does that help? $\endgroup$
    – Stephan Kolassa
    Commented May 15, 2017 at 6:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.