"You acted unwisely," I cried, "as you see
By the outcome." He calmly eyed me:
"When choosing the course of my action," said he,
"I had not the outcome to guide me."
(Ambrose Bierce: A Lacking Factor, from The Scrap Heap)
Great way to pose the question!
I'd like to offer a non-answer which presents the forecast problem from a different point of view.
As I see it, your way of presenting the problem portrays the statistician's goal as trying to build a "model" that's "close to the truth". Then one can speak of performance and pay.
This is a widespread way of viewing probability theory, but not the only one. I belong to those group of people (Jaynes, Jeffreys, Savage, de Finetti, and many others) who find it circular and ultimately hopeless.
The problem is that we cannot validate a forecast by comparing it to the "truth" – because we don't have the truth. If we had it, there would be no need of making forecasts. On the other hand, once we acquire knowledge of the truth, the forecast ceases to be important.
The point of a forecast is to try to guess the truth as best as possible given all information we have (and can gather). It may well happen that all info we have actually misses some crucial element, and therefore our forecast is grossly off the mark. Yet, that's the best we could do. It was impossible to use information that we didn't have – if we had had it, we would have used it! And maybe we didn't even know that some crucial info was missing. In fact, this is the whole crux of guessing and making forecasts – do the best with the info you have. We can't do our best with the info we don't have.
From this point of view, performance should be judged based on whether the information we actually had was optimally used. It may happen that the forecast was optimal, and yet quite off the mark; or vice versa, the forecast was poorly made, and yet turns out to be close to the truth.
Let me try to explain with an example.
Take a person $P$ in the population of your scenario.
Statistician $A$ examines the past history of $P$ and of $P$'s family (including, say, communications and interactions with other people) their general and mental health histories, and similar information. From this analysis, it appears that $P$ is a very pacific, altruistic, compassionate person, who would rather die than harm another person or let another person come to deadly harm, and who is generally loved by family, friends, acquaintances.
Statistician $A$ therefore gives an extremely low – but not zero – probability that $P$ would commit murder the next day.
Now comes statistician $B$. This statistician has exactly the same information as statistician $A$ – no more, no less.
Statistician $B$ gives 100% probability that $P$ will commit murder the next day. We can imagine that statistician $A$ asks $B$ why such a forecast, and statistician $B$ replies: "Because I hate $P$ – that person is meek and a good-doer; I can't stand people like that".
I imagine you'll agree with me that $B$'s reasoning and forecast are completely illogical and unreasonable. The reasoning and forecast of $A$, on the other hand, seem well-founded and reasonable. Or?
The next day comes. Arrived at work after the usual morning walk, person $P$ murders a colleague with a pair of scissors, then commits suicide.
Now, the statistician's $B$ forecast correctly "predicted" the murder. We could say that statistician $A$'s forecast was instead quite off the mark.
Should then a score function reward $B$ and penalize $A$? I want to remind you that $A$ and $B$ made their forecasts based on exactly the same information, summarized above.
We may wonder why $P$ committed murder – and the fact that we wonder confirms, in my opinion, the view that the murder was unexpected and $A$'s foracast was the most reasonable.
Here's an explanation. While person $P$ was walking on the street towards work in the morning, someone bumped into $P$ and purposely injected an allucinogen or some kind of neurology-altering drug. $P$ did not notice, maybe just checked for the wallet, in case the stranger was a pick-pocket. The stranger was actually an emissary for a secret lab that develops neurological weapons for a foreign country, and was in the nation just for the day, with the explicit purpose of testing the drug on a random citizen. The drug was designed to cause violent behaviour followed by self-violent behaviour.
I know this is a silly explanation, but you can find an alternative one of your own, maybe involving unsuspected congenital neurological problems or whatnot. The point is that unexpected events happen, and sometimes more than one in a row, as I'm sure you've experienced yourself in your life.
Yet, according to the "truth-based" reward/penalty point of view, such rare events will affect negatively a statistician who actually made a completely reasonable forecast. And I believe they shouldn't. (We cannot even exclude that many such events could happen.)
So my answer is that such the reward/penalty scheme should be based on whether the statistician does the most reasonable forecast given all the gatherable information, irrespective of whether the event happens or not. (Of course it's very difficult to come up with a score for this.)
Jaynes discusses this throughout his book, and in chapter 13 he quotes the passage by Bierce I put at the beginning, which makes the point brilliantly.