Equivalent of proper scoring rule for point forecasts

Question

Proper scoring rule is a concept used for evaluating density forecasts. What would be an equivalent for evaluating point forecasts? E.g. mean squared error seems like a proper metric for evaluating forecasts that target the expected value of the underlying random variable. This is because the forecast that truly minimizes the expected squared error (the population counterpart of the mean squared error) actually is the expected value. Meanwhile, mean absolute error does not seem proper for the same goal. On the other hand, it would seem proper when the target is the median of the underlying random variable. So what term is there to describe the propriety of a metric for evaluating point forecasts?

Answer 1

As mentioned by QMath, the term (strictly) consistent scoring function is used in the statistical literature to describe functions which evaluate point predictions based on the same principles as (strictly) proper scoring rules.

The (slightly simplified) definition uses a set $A$ of possible point forecasts, a set $O$ of observations, a set $\mathcal{F}$ of distributions, and a statistical property/functional $T: \mathcal{F} \to A$. Examples of functionals are the mean or the median, as already mentioned in the question. A scoring function is a function $S : A \times O \to \mathbb{R}$. It is consistent for $T$ if $$ \mathbb{E}_{Y\sim F} \, S(T(F), Y) \le \mathbb{E}_{Y\sim F} \, S(x, Y) $$ for all $x \in A$ and $F \in \mathcal{F}$. It is strictly consistent if equality holds if and only if $x=T(F)$. The fact that $T(F)$ is a minimizer is of course simply convention.

The idea behind this is (analogous to proper scoring rules) that the point forecast $T(F)$ resulting from the true underlying distribution of $Y$ will receive the lowest score/loss on average. Similar to their proper scoring rule counterpart, they enable forecast rankings and can be used to do regression or train ML models, for a desired statistical property. The most popular examples are squared error (for the mean) and absolute error (for the median).

Two additional points relating to the discussion below the question:

1. Proper scoring rules are not restricted to density forecasts. They assess probabilistic predictions. Depending on how you deal with your distributions, you can use different scoring rules. For instance, you can define them for the CDF or even the characteristic function.
2. Tim stated above that "with point predictions, you are directly predicting the values". The theory behind scoring functions sees this slightly different. There we have some distribution of the target in mind and for some reason, we have to reduce this to a single point, e.g. the mean. So if a value $x$ is reported, this is not a statement, that the forecaster believes, that the observation will be $x$. It is a piece of statistical information. This becomes obvious when dealing with discretely distributed data: A forecaster can report $x=3.5$ here since they want to report the mean, but for a forecaster who wants to predict the observation, a non-integer forecast makes no sense at all.

For more details, see Making and evaluating point forecasts especially Section 3.2. (The definition there is for set-valued forecasts, so it is more general, but less intuitive, I think)