Initially, I had only heard of MLE and use it for almost everything, e.g. point estimate and model selection (with some penalty).

Then, MSE appeared, which seems to play the same role as MLE does. I learned that both MSE and MLE are results from loss functions (entropy vs quadratic). Choose which one to use is completely up to one's belief.

Now, I came across stuff called scoring rule. There are new terms like Logarithmic scoring function and Brier scoring function. However, I feel they are just another names of MLE and MSE methodology. Minimize Brier is essentially minimizing MSE, and Maximize Logarithmic is exactly the same as MLE.

Am I right here? Why do we have different names for the exact same things between scoring rule and loss function?


All of these terms are hierarchically nested. From Adrian E. Raftery and Tilmann Gneiting, "Strictly Proper Scoring Rules, Prediction and Estimation" Nov. 2005:

In estimation problems, strictly proper scoring rules provide attractive loss and utility functions that can be tailored to a scientific problem. To fix the idea, suppose that we wish to fit a parametric model $P_\theta $ based on a sample $X_1,...,X_n$. To estimate $\theta$, we might measure the goodness-of-fit by the mean score $$\mathcal{S}_n(\theta)=\frac{1}{n}\sum_{i=1}^{n}S(P_\theta,X_i),$$ where $S$ is a strictly proper scoring rule. If $\theta_0$ denotes the true parameter, asymptotic arguments indicate that $\text{arg max}_\theta\mathcal{S}_n(\theta)\to\theta_0$ as $n\to\infty$. This suggests a general approach to estimation: choose a strictly proper scoring rule that is tailored to the scientific problem at hand, maximize $\mathcal{S}_n(\theta)$ over the parameter space, and take $\hat{\theta}_n=\text{arg max}_\theta\mathcal{S}_n(\theta)$ as the optimum score estimatior based on the scoring rule $S$... Maximum likelihood estimation forms a special case of optimum score estimation, and optimum score estimation forms a special case of $M$-estimation, in that the function to be optimized derives from a strictly proper scoring rule.

So we can see that in general, we have a very flexible mechanism to estimating $\hat{\theta}$ by means of choice of $S$ and that MLE is just one particularly well-studied scoring rule within this framework.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.