Evaluating probabilistic forecasts of K-most-likely events from an arbitrarily large event space

Suppose a populous nation has a high homicide rate and an understaffed police force. The police chief hires a statistician and together they decide to take a preventative approach by identifying would-be-murderers before they commit the crime, along the lines of Minority Report.

The police chief requires the statistician to provide the following on a daily basis:

1. A list of tomorrow's top 100 most-likely murderers. (The statistician may have information about the entire citizen population, but the chief doesn't have time to think about more than 100 cases.)
2. For each person on the list, the statistician's best estimate of the probability that the person will commit a murder (in the absence of intervention).

The police chief will regularly evaluate the statistician's forecasts and provide bonus pay for good performance. Unfortunately, the chief does not know how to score the forecasts in a way that incentivizes the statistician to honestly strive toward the objectives (1) and (2). Can you help?

Here are two basic proposals of increasing complexity:

• Score = recall = The number of people who attempt murder that the statistician included on the list. But this gives no incentive for accurate probabilities (2).
• Score = $100 - \sum_{i=1}^{100} (O_i - p_i)^2$, similar to the Brier score. Here $p_i$ is the forecasted probability for the $i$th person on the list, and $O_i$ is the true outcome (0 or 1) for their murdership status. But the statistician can easily maximize this by selecting 100 people with no chance of being murderers and taking $p_i$ to be identically 0.

Any other ideas? I strongly suspect that this is not a new problem; a good reference may suffice.

• When computing the score, is it allowed to use the outcomes of people that are not on the list? – Fabian Sep 18 '15 at 9:58
• @Fabian, no. So a high score could correspond to low recall if the number of murderers is much greater than 100, and I'm OK with that. One possible reason for my choice is that this nation might not even have a census; the statistician might be using ad-hoc methods to identify potential murderers without even knowing what is the total population of the nation. – zkurtz Sep 18 '15 at 10:49

Indeed, there are many other ideas! If you call $p$ the estimated probability and $y$ the output, the following metrics are widely used:

• MAE = $\sum_i|p_i-y_i]$
• MSE = $\sum_i(p_i-y_i)^2$,
• logarithmic loss = $\sum_i(1-y_i)\log p_i+y_i\log (1-p_i)$

In these examples, a misclassification is more penalized if the associated probability is high.

Note that these approaches will be affected if you replace $p$ by $p^2$. Besides, it is hard to say if if a logarithmic loss of 0.34 is high or low - whereas an error rate of 95% is self explanatory.

A way to circumvent this is to use the AUC, which lies between 0 and 1 and focuses on the rank of the proposed probabilities.

• AUC $=\frac{S_0-n_0(n_0+1)/2}{n_0n_1}$

Where $n_0$ and $n_1$ are the number of positive and negative examples. $S_0$ is the sum of the ranks of the positive examples.

More details about AUC can be found here: http://home.cse.ust.hk/~qyang/Teaching/537/Papers/AUC-evaluation.pdf

• The MAE and MSE clearly suffer from the same flaw as the Brier score as I pointed out. I'm not sure how you define the logarithmic loss when p = 0. It could be worth digging deeper into whether the AUC is compatible with (1) and (2) in the OP. – zkurtz Sep 21 '15 at 16:26

There has, as you suggest, been quite a lot of discussion in various communities surrounding this point. The crux of the problem is that when you have a relatively rare event (the probability that even the most murderous person commits a murder on a given day has to be quite small- naively, for someone who commits murders on 36 days a year, it would be only ~.1 for a given day) evaluating the value of that prediction presents significant challenges. A correct prediction that the probability is .1 still only results in a 10 percent chance of the event occuring.

Luckily, there are many branches of inquiry which concentrate on rare events.

Meterology, for example, consists almost entirely of predicting relatively rare events. In this paper (1) , Marzban evaluates several metrics for evaluating models similarly in terms of the propensity of those metrics to induce the forecaster to over- or underestimate the likelihood of a given rare event.

While there are many specific model variations that are discussed in more detail in the paper, the general approaches are:

1. Some combination of the false alarm rate (0 results classified as 1 predictions) and the probability of detection (1 results classified as 0 predictions)
2. Critical Success Index
3. Skill scores
4. Custom angle metrics (Marzban 754-55)

The general results we see for a model is that some models encourage over-predicting the rare event in question and some encourage under-predicting the event.

In this case, the police chief would want to make a decision on which metric to employ to evaluate the results of the statistician's model informed by the way in which the prediction would be used. If it was, as you mention, in Minority Report, going to be used to preemtively imprison people, presumably we would want to encourage the statistician to under-predict the rare events (or, in a more authoritarian state, over-predict them into near-oblivion).

(1) Caren Marzban, 1998: Scalar Measures of Performance in Rare-Event Situations. Wea. Forecasting, 13, 753–763.

• The rareness of events in my example is not a primary point of my question. Please focus on the criteria (1) and (2). I.e., K is fixed (for K-most-likely) and the statistician is asked to report honest (or, say "median" or "modal" for bayesian approaches) estimates of the probabilities, NOT taking into account how the predictions would be used. – zkurtz Sep 21 '15 at 16:23
• I understood the question and both points. K is fixed, but p-values would presumably be low, even for the most likely of murderers. The paper shows how it's a trade-off to use one metric or another, so regardless of how the police chief chooses to score the predictions, the statistician can game them for profit one way or another. if you're looking to incentivize an actual statistician, you have to choose. – Thomas Cleberg Sep 21 '15 at 16:40
• "you have to choose" is a very interesting claim that would be answer-worthy if you can prove it. Be aware that it is not generally a true claim; see the concept of a 'proper scoring rule': en.wikipedia.org/wiki/Scoring_rule#Proper_scoring_rules – zkurtz Sep 21 '15 at 16:52
• To be clear, when the entire population (rather than just the top K) of probabilities, the Brier score is indeed proper. I'm asking for a proper scoring rule when just the top K probabilities are reported; if it does not exist, I'd like a proof or at least strong argument of nonexistence. – zkurtz Sep 21 '15 at 16:54
• Re: Marzban reference. Note they treat only the case in which "both the observations and the forecasts are assumed to be binary". The forecasts in the OP are probabilistic, not binary. – zkurtz Sep 21 '15 at 17:22

"You acted unwisely," I cried, "as you see
By the outcome." He calmly eyed me:
"When choosing the course of my action," said he,
"I had not the outcome to guide me."

(Ambrose Bierce: A Lacking Factor, from The Scrap Heap)

Great way to pose the question!

I'd like to offer a non-answer which presents the forecast problem from a different point of view.

As I see it, your way of presenting the problem portrays the statistician's goal as trying to build a "model" that's "close to the truth". Then one can speak of performance and pay.

This is a widespread way of viewing probability theory, but not the only one. I belong to those group of people (Jaynes, Jeffreys, Savage, de Finetti, and many others) who find it circular and ultimately hopeless.

The problem is that we cannot validate a forecast by comparing it to the "truth" – because we don't have the truth. If we had it, there would be no need of making forecasts. On the other hand, once we acquire knowledge of the truth, the forecast ceases to be important.

The point of a forecast is to try to guess the truth as best as possible given all information we have (and can gather). It may well happen that all info we have actually misses some crucial element, and therefore our forecast is grossly off the mark. Yet, that's the best we could do. It was impossible to use information that we didn't have – if we had had it, we would have used it! And maybe we didn't even know that some crucial info was missing. In fact, this is the whole crux of guessing and making forecasts – do the best with the info you have. We can't do our best with the info we don't have.

From this point of view, performance should be judged based on whether the information we actually had was optimally used. It may happen that the forecast was optimal, and yet quite off the mark; or vice versa, the forecast was poorly made, and yet turns out to be close to the truth.

Let me try to explain with an example.

Take a person $$P$$ in the population of your scenario.

Statistician $$A$$ examines the past history of $$P$$ and of $$P$$'s family (including, say, communications and interactions with other people) their general and mental health histories, and similar information. From this analysis, it appears that $$P$$ is a very pacific, altruistic, compassionate person, who would rather die than harm another person or let another person come to deadly harm, and who is generally loved by family, friends, acquaintances.

Statistician $$A$$ therefore gives an extremely low – but not zero – probability that $$P$$ would commit murder the next day.

Now comes statistician $$B$$. This statistician has exactly the same information as statistician $$A$$ – no more, no less.

Statistician $$B$$ gives 100% probability that $$P$$ will commit murder the next day. We can imagine that statistician $$A$$ asks $$B$$ why such a forecast, and statistician $$B$$ replies: "Because I hate $$P$$ – that person is meek and a good-doer; I can't stand people like that".

I imagine you'll agree with me that $$B$$'s reasoning and forecast are completely illogical and unreasonable. The reasoning and forecast of $$A$$, on the other hand, seem well-founded and reasonable. Or?

The next day comes. Arrived at work after the usual morning walk, person $$P$$ murders a colleague with a pair of scissors, then commits suicide.

Now, the statistician's $$B$$ forecast correctly "predicted" the murder. We could say that statistician $$A$$'s forecast was instead quite off the mark.

Should then a score function reward $$B$$ and penalize $$A$$? I want to remind you that $$A$$ and $$B$$ made their forecasts based on exactly the same information, summarized above.

We may wonder why $$P$$ committed murder – and the fact that we wonder confirms, in my opinion, the view that the murder was unexpected and $$A$$'s foracast was the most reasonable.

Here's an explanation. While person $$P$$ was walking on the street towards work in the morning, someone bumped into $$P$$ and purposely injected an allucinogen or some kind of neurology-altering drug. $$P$$ did not notice, maybe just checked for the wallet, in case the stranger was a pick-pocket. The stranger was actually an emissary for a secret lab that develops neurological weapons for a foreign country, and was in the nation just for the day, with the explicit purpose of testing the drug on a random citizen. The drug was designed to cause violent behaviour followed by self-violent behaviour.

I know this is a silly explanation, but you can find an alternative one of your own, maybe involving unsuspected congenital neurological problems or whatnot. The point is that unexpected events happen, and sometimes more than one in a row, as I'm sure you've experienced yourself in your life.

Yet, according to the "truth-based" reward/penalty point of view, such rare events will affect negatively a statistician who actually made a completely reasonable forecast. And I believe they shouldn't. (We cannot even exclude that many such events could happen.)

So my answer is that such the reward/penalty scheme should be based on whether the statistician does the most reasonable forecast given all the gatherable information, irrespective of whether the event happens or not. (Of course it's very difficult to come up with a score for this.)

Jaynes discusses this throughout his book, and in chapter 13 he quotes the passage by Bierce I put at the beginning, which makes the point brilliantly.