# Is Brier score strictly proper in multi-label problems?

In problems where one of $$3+$$ categories can be observed and we prodict the probability of each category being observed, it is known that the Brier score is a strictly proper scoring rule that is uniquely optimized in expected value by the true probability values [1, 2, 3, 4]. The machine learning community often refers to these problems with $$3+$$ categories as "multiclass" problems.

In contrast, "multi-label" problems allow for all, none, or any combination of categorical outcomes to be observed, and we model the probability of each individual outcome, possibly with relationships between the outcomes (e.g., if there is a horse in a photo, there probably isn't an airplane).

For a multi-label outcome, is the Brier score still a strictly proper scoring rule?

• I'm sure I'll post the same question about log loss (which I would link here). If an answer here wants to address both, that would not be unappreciated (especially if there is an interesting reason why the answers differ).
– Dave
Commented Nov 20, 2023 at 15:11
• "it is known that the Brier score is a strictly proper scoring rule that is uniquely maximized in expected value by the true probability values". Reading the wikipedia definition here gives me a largely different impression. Suppose my sample is $x = [1,1,1,0,0,0]$ drawn iid from Bernoulli 0.5 RV. The Brier score is "uniquely maximized" if it is exactly $Q = [1,1,1,0,0,0]$, not $[0.5, 0.5, 0.5, 0.5, 0.5, 0.5]$ or "the true probability values" as you say. Commented Nov 20, 2023 at 15:16
• @AdamO I have included links discussing why Brier score is a strictly proper scoring rule.
– Dave
Commented Nov 20, 2023 at 15:23
• @AdamO What would you say defines a strictly proper scoring rule? What I'm saying seems to come straight from the Gneiting/Raftery 2007 JASA paper.
– Dave
Commented Nov 20, 2023 at 16:33
• I think my answer to this question explains how quadratic loss/Brier score is (strictly) proper in the multi-label setting. Maybe you mean some specific situation which was not addressed there? Commented Nov 21, 2023 at 8:09