I am using Brier score as my scoring metric, as opposed to log loss. My reason for doing so was because I read it was more focused on the positive class than log loss. But how is it?
https://machinelearningmastery.com/tour-of-evaluation-metrics-for-imbalanced-classification/
The full quote from your link is:
The benefit of the Brier score is that it is focused on the positive class, which for imbalanced classification is the minority class. This makes it more preferable than log loss, which is focused on the entire probability distribution.
The Brier score is calculated as the mean squared error between the expected probabilities for the positive class (e.g. 1.0) and the predicted probabilities. ...
BrierScore = 1/N * Sum i to N (yhat_i – y_i)^2
The formula is correct. However, the explanation above it is wrong.
The Brier score is not only calculated over positive class instances. Rather, it is calculated over all instances, positive and negative. So if your outcomes can be $y=0$ or $y=1$, we can write the score as
$$ \frac{1}{N}\sum_{i=1}^n (\hat{y}_i-y_i)^2 = \frac{1}{N}\bigg(\sum_{y_i=0} \hat{y}_i^2 + \sum_{y_i=1} (\hat{y}_i-1)^2\bigg). $$
We see the contributions of both "negative" instances (with $y_i=0$, and larger $\hat{y}_i$ lead to higher loss) and "positive" instances (with $y_i=1$, and smaller $\hat{y}_i$ lead to higher loss).
(Note that $\hat{y}$ is assumed to be a probabilistic prediction, which your link does IMO not emphasize enough.)
As a matter of fact, the Brier score is a proper scoring-rule, just like the log loss. So both will be minimized in expectation by correct probabilistic predictions, i.e., when $P(y_i=1)=\hat{y}_i$. And if you use probabilistic predictions and proper scoring rules, all problems with unbalanced data vanish.
