4
$\begingroup$

Basic question. To solve the $-\infty$ one find in the logarithm of zero a common transformation is the log plus one transformation. So I applied it to the Logarithmic Scoring Rule and got the following rule, normalized by $log(2)$ with $c$ being the actual probability and $p$ the predicted probability:

$$S(c,p)=\frac{c \cdot log(p+1)+(1-c) \cdot log(2-p)}{log(2)}$$

I tested it with many values for $c$, and if I understand right, the following should always hold:

$$\underset{p}{\mathrm{argmax}}\space \left\{S(c,p)\right\} = c$$

So, is $S$ a proper scoring rule?

$\endgroup$

1 Answer 1

5
$\begingroup$

No, it isn't.

Remember the definition of a proper scoring rule: it is a loss function of the actual outcome and your predictive density (in your case of a binary classification, the predictive density is just an estimated $p$), which is minimized in expectation by the true future density.

(So, to be in exact accordance with the definition of a scoring rule, $S$ should be a function of an observed outcome, not the unobservable probability $c$ - but $c$ will come in when you average over many observed outcomes, so it doesn't really matter.)

Note that a proper scoring rule needs to be minimized, not maximized, so your argmax is irrelevant.

However, even putting a minus sign in front of your rule won't help. We can simply calculate the derivative of $S$ with respect to $p$ and check whether it is zero if $p=c$:

$$ \frac{\partial S}{\partial p} = \frac{1}{\log 2}\bigg(\frac{c}{p+1}-\frac{1-c}{2-p}\bigg),$$

so

$$ \frac{\partial S}{\partial p}(c,c) = \frac{1}{\log 2}\bigg(\frac{c}{c+1}-\frac{1-c}{2-c}\bigg),$$

which is nonzero for $c\neq\frac{1}{2}$. In fact, the partial derivative above is zero if and only if

$$ p = \frac{1-3c}{c-3} $$

(barring any errors I made), which is usually not solved for $p=c$ and will actually be negative if $c<\frac{1}{3}$.

Finally, you can plot $S$ as a function of $p$ for given $c$. For instance, for $c=\frac{1}{4}$:

non-scoring rule

SS <- function ( cc, pp ) (cc*log(pp+1)+(1-cc)*log(2-pp))/log(2)

cc <- 0.25
pp <- seq(.01,.99,by=.01)
plot(pp,SS(cc,pp),type="l",xlab="",ylab="")

Again, this is not minimized for $p=c=\frac{1}{4}$.

$\endgroup$
1
  • $\begingroup$ And I see my tests with different values of $c$ were actually bogus. Oh well, thanks for the reminder! $\endgroup$
    – Firebug
    May 23, 2016 at 11:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.