3
$\begingroup$

Brier score can be computed for joint predictions of multiple variables, each with multiple categories. Let's say we have 4 variables with 3 possible classes each.

In that case, the denominator of the Brier score is 4 (as it seems implied in the original 1956 Brier paper) or 12 (as seems implied in Wikipwdia "and N the overall number of instances of all classes.") ?

Is there a value of the Brier score that is, by convention (as certan values of the p-value ), considered good enought?

$\endgroup$

1 Answer 1

4
$\begingroup$

I am not aware of a standard for this denominator. However, I would strongly lean towards using the number of variables, 4. This way, your overall result is scaled between 0 and 1, with perfect predictions achieving 0 (assuming a negative "smaller is better" orientation). Since every separate variable contributes something between 0 and 1, dividing by 12 would have you end up with something between 0 and 0.333. See also How to Compute the Brier Score for more than Two Classes.

As to a Brier score that is "good enough", no, that does not exist in isolation, for the same reasons that any other outputs of statistical models - parameter estimates or predictions - do not have notions of "good enough" in isolation. Best to compare your model to a benchmark model, rather than a benchmark score. See:

$\endgroup$
1
  • $\begingroup$ Thank you, I did read that question, but it uses there only one variable, so I wasn't sure. By the way, when computed in that way, the maximum value is 2. $\endgroup$
    – Antonello
    Apr 20 at 14:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.