Suppose I have a confusion matrix, or alternatively any one or more of accuracy, sensitivity, specificity, recall, F1 score or friends for a binary classification problem.
How can I calculate the Brier or log score of my classifications from this information?
You can't.
The Brier score or log score are calculated from probabilistic classifications and corresponding outcomes. The confusion matrix, accuracy etc. are calculated from hard 0-1 classifications and the corresponding outcomes. If you have probabilistic classifications, you can turn them into hard ones by using a threshold, but since that threshold cannot be trained, it is an absolutely crucial ingredient in calculating the confusion matrix etc.:
$$ \text{Brier or log score} = f(\hat{p}, y) $$
versus
$$\text{Confusion matrix, accuracy etc.} = g(\hat{y}, y) = g\big(\hat{y}(\hat{p},t),y\big), $$
where $\hat{p}$ is a vector of predicted class membership probabilities, $y$ is a 0-1 vector of the corresponding true class memberships, $0\leq t\leq 1$ is a threshold, and $\hat{y} = \hat{y}(\hat{p},t)$ is the dichotomization of probabilistic predictions into hard 0-1 classifications using the threshold $t$ (relatedly, see Frank Harrell on dichotomania).
Alternatively, $\hat{y}$ might be directly output by a classifier that does not (explicitly) use probabilitites, in which case we can't even calculate the Brier or log score. (In my opinion, that is a strong argument against such classifiers.)
Suppose we have no useful predictors distinguishing the instances in our evaluation sample. For instance, these data points might all have the exact same predictor data. Alternatively, we can apply the argument below separately for each possible combination of predictor values.
Thus, as far as our model knows, all instances have the same probability $\hat{p}$ of belonging to the target class. In particular, this means that for a "hard" classification, all instances will be classified as either 0 or 1.
Let's assume our model is correct in this, and all instances really do have the same true probability $p$ of belonging to the target class. We will assume that $p=0.1$.
We consider four different possible probabilistic predictions - perhaps they come from four different models: $\hat{p}\in\{0.05, 0.10, 0.20, 0.30\}$. Per above, in order to get hard classifications to calculate the confusion matrix etc., we need a threshold. Let's say we use $t=0.15$.
Then if $\hat{p}=0.05$ or $\hat{p}=0.10$, both of which are below the threshold, all instances will be classified as 0, whereas for $\hat{p}=0.20$ or $\hat{p}=0.30$, all instances will be classified as 1. Thus, the confusion matrix, accuracy etc. will be identical for all $\hat{p}<t$ and for all $\hat{p}>t$, regardless of the actual outcome.
But of course the Brier and the log scores will be different for different $\hat{p}$. Their specific values will depend on the outcome we actually observe (just as the confusion matrix etc.), but given the observed outcome, the scores will depend on $\hat{p}$. For instance, the expected scores are:
$$ EB(\hat{p}=0.05) = 0.0925, \quad EB(\hat{p}=0.10) = 0.09, \quad EB(\hat{p}=0.20) = 0.1, \quad EB(\hat{p}=0.3) = 0.13 $$
and
$$ E\ell(\hat{p}=0.05) = 0.35, \quad E\ell(\hat{p}=0.10) = 0.33, \quad E\ell(\hat{p}=0.20) = 0.36, \quad E\ell(\hat{p}=0.30) = 0.44. $$
(Note that both scores are minimized in expectation by the true probability $\hat{p}=p$, because that is the definition of their being proper, and they are actually strictly proper.)
Thus, since the confusion matrix, accuracy etc. are constant with respect to $\hat{p}$ above and below the threshold $t$, whereas the Brier and the log score are not constant, you cannot derive the Brier or log score from the confusion matrix, accuracy etc.
Note finally that this argument does not depend on $p$ being different from $0.5$, i.e., on having "unbalanced" data.
A point raised in some related conversations that seem to have inspired the posting of this question is that a confusion matrix can contain probability values if you normalize by the sum of the entries.
$$ \begin{pmatrix} 4 & 2\\ 1 & 3 \end{pmatrix} \overset{\div 10}{\rightarrow} \begin{pmatrix} 0.4 & 0.2\\ 0.1 & 0.3 \end{pmatrix}\\ \begin{pmatrix} 8 & 2\\ 3 & 7 \end{pmatrix} \overset{\div 20}{\rightarrow} \begin{pmatrix} 0.4 & 0.1\\ 0.15 & 0.35 \end{pmatrix} $$
However, notice that each of these matrices contains four values, while they correspond to ten and twenty classification attempts, respectively. This means there is a mismatch between the number of true $y_i$ and predicted $\hat p_i$ when you go to calculate something like Brier score or log-loss (even c-index or ROCAUC) that sums over the number of observations (ten and twenty in the two respective matrices). Consequently, even this normalization of the confusion matrices does not allow for calculations of the Brier score or log loss, giving further evidence that the confusion matrix is not the whole story. Since the accuracy, sensitivity, specificity, $F_1$, $F_{\beta}$, precision, recall, FPR, FNR, TPR, TNR, PPV, NPV, Matthews correlation coefficient, and all of the other friends mentioned here are derived from the confusion matrix, those would not tell the whole story, either.
