Let $X_i\sim B(\pi_i), \text{for }i=1,2,\cdots,n$. I have two models and I want to compare which of them forecast better.
Model 1: Estimates the parameters with maximum likelihood.
Model 2: Estimates the parameters with Bayes.
I use the Brier score and the Logarithmic scoring rule for comparison. The results are:
>> Model 1: 0.2505 (Brier), 0.6350 (minus log-score)
>> Model 2: 0.2544 (Brier), 0.6028 (minus log-score)
The smaller the score, the better the model. So, according to Brier Score Model 1 is better, and according to log-score Model 2 is better.
I would like to ask, why there is this difference. Also, is there a paper for learning how to compare the forecasting ability of a frequentist model with a Bayesian one?