What is null distribution of brier's score or logarithmic score? How to test them properly? I would like to get some p-values for predictions in an independent test set according to proper scoring rules. So I would like to know if my brier score/logarithmic score is statistically significantly better than what would I expect by chance. 
In a balanced situation, if a model always predicts only 0.5 probability, I would get brier score 0.25 and a logarithmic score of 0.69. However, if I run a permutation test, i.e. I shuffle my predictions and recompute the scores again, the mean scores of this null distribution are around 0.37 and 1.1 for brier's score and logarithmic score respectively, which is much worse than just a model that always predicts 0.5. 
What is a null distribution of these scores and how to appropriately statistically test it?
 A: This is an interesting question. Unfortunately, I am not aware of any published research on this, so I will give some thoughts off the top of my head below.
The (perfectly) balanced situation
Unfortunately, null hypothesis significance testing (NHST) and proper scoring rules don't work well together in the perfectly balanced situation.
If your classes are perfectly balanced, and the null hypothesis is that the population proportions (of $p=\frac{1}{2}$) are the best probabilistic classifier you can have, then every instance, from each of the two balanced classes, will add the same contribution to your scores: $p^2=\frac{1}{4}$ to the Brier score, and $\log p=-\log 2\approx -0.69$ to the log score.
Thus, under the null hypothesis, the Brier and the log scores are degenerate distributions: point masses at $\frac{1}{4}$ and $-\log 2$, respectively. Consequently, any non-constant probabilistic prediction, which will have a Brier or log score which differs from this number, will lead to a rejection of the null hypothesis - whether one- or two-sidedly.
The (even slightly) unbalanced situation
If you have a population proportion that even slightly deviates from perfect balance, $p\neq\frac{1}{2}$, things change. Under this "climatological" or "population" null model, an instance of the majority class will contribute $p^2$ to the Brier and $\log p$ to the log score, and an instance of the minority class will contribute $(1-p)^2$ and $\log(1-p)$, respectively. Thus, if you have $n$ instances in your test set, the total Brier and log scores follow a scaled binomial distribution, where you don't add $1$ with probability $p$ and $0$ with probability $1-p$, but $p^2$ and $(1-p)^2$ (for the Brier score) and $\log p$ and $\log(1-p)$ for the log score. Now we can calculate critical regions and reject the null hypothesis if our non-constant model yields scores in the critical region. A normal approximation may well be useful.
But we don't know whether we have balance!
Now, the discussion above presumes that we know whether our situation is balanced or not. But we usually don't know that, because we are estimating our population proportion $p$ from the data itself! (I can't think of a planned experiment, where we purposely do balanced sampling, but would afterwards assume a balanced population to assess a classifier.) And I would presume that the potential variability in this estimate $\hat{p}$ might definitely have a major impact on the null distribution of our scores.
Resampling
Here is what I would do.

*

*Take a bootstrap sample from the training data. Estimate the population $p$ from this sample as $\hat{p}_i$.

*Take a bootstrap sample from the test data. Apply the constant prediction $\hat{p}_i$ to this sample, and record the Brier and/or log scores $BR_i$ and $L_i$.

*Repeat this process many times.

*Use the resulting $(BR_i)$ and $(L_i)$ as empirical null distributions.

Notes

*

*This approach is also applicable to the multi-class situation, or to probabilistic predictions for continuous outcomes which are then assessed using proper scoring rules.


*You may run into the situation where the bootstrap sample from the training set does not contain a class, which is then present in the bootstrap sample from the test set. Then you will try to calculate $\log 0$ and get an error. This can be considered a bug or a feature.
I personally would simply discard such situations, reasoning that since we are interested in a classification with the classes we are considering, any "reasonable" null distribution should contain samples from all classes - and a resampled distribution that does not do is is therefore not useful. Yes, this argument can be debated.


*If you are doing time series forecasting, the bootstrapping from the training data can be done, but one should note that this amounts to positing no temporal dynamics under the null distribution (the "climatological forecaster"). If one prefers a more nuanced null distribution, e.g., one that takes seasonality into account, one would need to be a little more sophisticated, e.g., by doing a stratified bootstrap.
