# Is the following a proper scoring rule?

Basic question. To solve the $-\infty$ one find in the logarithm of zero a common transformation is the log plus one transformation. So I applied it to the Logarithmic Scoring Rule and got the following rule, normalized by $log(2)$ with $c$ being the actual probability and $p$ the predicted probability:

$$S(c,p)=\frac{c \cdot log(p+1)+(1-c) \cdot log(2-p)}{log(2)}$$

I tested it with many values for $c$, and if I understand right, the following should always hold:

$$\underset{p}{\mathrm{argmax}}\space \left\{S(c,p)\right\} = c$$

So, is $S$ a proper scoring rule?

No, it isn't.

Remember the definition of a proper scoring rule: it is a loss function of the actual outcome and your predictive density (in your case of a binary classification, the predictive density is just an estimated $p$), which is minimized in expectation by the true future density.

(So, to be in exact accordance with the definition of a scoring rule, $S$ should be a function of an observed outcome, not the unobservable probability $c$ - but $c$ will come in when you average over many observed outcomes, so it doesn't really matter.)

Note that a proper scoring rule needs to be minimized, not maximized, so your argmax is irrelevant.

However, even putting a minus sign in front of your rule won't help. We can simply calculate the derivative of $S$ with respect to $p$ and check whether it is zero if $p=c$:

$$\frac{\partial S}{\partial p} = \frac{1}{\log 2}\bigg(\frac{c}{p+1}-\frac{1-c}{2-p}\bigg),$$

so

$$\frac{\partial S}{\partial p}(c,c) = \frac{1}{\log 2}\bigg(\frac{c}{c+1}-\frac{1-c}{2-c}\bigg),$$

which is nonzero for $c\neq\frac{1}{2}$. In fact, the partial derivative above is zero if and only if

$$p = \frac{1-3c}{c-3}$$

(barring any errors I made), which is usually not solved for $p=c$ and will actually be negative if $c<\frac{1}{3}$.

Finally, you can plot $S$ as a function of $p$ for given $c$. For instance, for $c=\frac{1}{4}$:

SS <- function ( cc, pp ) (cc*log(pp+1)+(1-cc)*log(2-pp))/log(2)

cc <- 0.25
pp <- seq(.01,.99,by=.01)
plot(pp,SS(cc,pp),type="l",xlab="",ylab="")


Again, this is not minimized for $p=c=\frac{1}{4}$.

• And I see my tests with different values of $c$ were actually bogus. Oh well, thanks for the reminder! May 23, 2016 at 11:00