How to rate successive predictions of the outcome of an event which are made while it is taking place?

Question

Consider the scenario of making successive predictions about the outcome of a sporting event while it is taking place. I will use tennis as a concrete example because it has clearly defined moments at which it is sensible to make a new prediction (i.e. after every point is played) but I am also interested in cases such as soccer where predictions can be made with arbitrary frequency (e.g. every five seconds).

After every point of the tennis match is played, a model makes a prediction giving the probability that each player will go on to win the match. In a match with $N$ points played, $N$ predictions are made and my question is how should the performance of these $N$ predictions be summarised to give a single value for the performance of the model on this match?

An obvious starting point would be the Brier Score, i.e. the average squared error of the predictions. However, taking this approach may not adequately penalise outliers - e.g. a single terrible prediction among otherwise good ones. At the other extreme, the maximum absolute error might favour a consistently mediocre set of predictions over a set of very good predictions and one terrible prediction.

Which of these possibilities is more tolerable is open to debate but I seem to have described a general problem with summary statistics. What I am more interested in learning about are approaches that are specifically tailored to my scenario, i.e. which take into account the fact that these predictions are all made on the same outcome and each prediction is made with successively more information.

Answer 1

What I am more interested in learning about are approaches that are specifically tailored to my scenario, i.e. which take into account the fact that these predictions are all made on the same outcome and each prediction is made with successively more information.

That's the key: for your predictor to be acceptable, it should get better the closer we get to the end of the match, because it uses more and more information. To apply a modified Brier-score logic, let $o_F$ be a binary $\{0,1\}$ representing the final outcome of the game (say "$1$" = player A wins), $I_k$ be the set containing information available up to and including events as of stage $k$ of the game, and let $f_k(o_F=1\mid I_k)$ be the predicted probability that player A will win, given this information. Then we can define a "cumulative" Brier-like score as

$$BS_k = \frac 1k\sum_{i=0}^k \Big(f_i(o_F=1\mid I_i) - o_F\Big)^2 \qquad k=0,...,N$$

(I have included ${k=0}$ to cover the prediction before the game starts). Then, a reasonable demand for a good predictor is that the sequence $\{BS_k\}$ be decreasing. Comparing two competing predictors would amount to compare their rates of decrease.

You could easily try also a "moving window" expression, where some past information gets discarded if it becomes "old enough" - it depends on what information you will deem relevant in predicting the outcome, and will eventually include as input to your predictor.

Of course, if your predictors are human beings, you don't need to find out their prediction functions - you will just record their predictions and compare them.