Questions tagged [scoring-rules]
Scoring rules are used to assess the accuracy of predicted probabilities, or more generally of predictive densities. Examples of scoring rules include the logarithmic, Brier, spherical, ranked probability and the Dawid-Sebastiani score and the predictive deviance.
256
questions
2
votes
2
answers
54
views
Writing Kullback–Leibler divergence in terms of the score function
Assume we have two density functions $p(x)$ and $p'(x)$ for $x\in R^d$.
I would find a connection between Kullback–Leibler divergence between two densities in terms of the difference between the ...
- 61
5
votes
1
answer
350
views
What is the best epoch to evaluate the test images?
I created a training, a validation and a test set for an image classification task. Then, I did training using the training and did evaluation on validation set. So, the next step is to evaluate the ...
- 53
0
votes
0
answers
25
views
Comparing performance of probabilistic regression models - how to adapt Brier score?
Suppose I have two predictions models, Model 1 and Model 2. I have a dataset containing observations, features and actual outcomes. For each observation, the “outcomes” (i.e. predictions) that the ...
- 101
2
votes
1
answer
37
views
Evaluating estimator of expected value plus variation
I know that for a typical, the estimator can be evaluated based on the mean squared error (MSE) of the predictions. How can I evaluate an estimator that instead gives a value that is the prediction ...
- 23
10
votes
1
answer
160
views
There are infinitely many proper scoring rules. Are they all equally valid? Or is log loss superior because of its connection to max likelihood?
I'm kind of obsessed with binary loss functions.
How to create a (binary) loss function (scoring rule):
Create a function, $f: [0,1] \rightarrow \mathbb{R}_{\geq 0}
$, that is symmetric about $x=\...
0
votes
0
answers
43
views
Derivative of the multivariate normal cumulative distribution function (CDF) with reparameterisation [duplicate]
I would like to learn how to calculate the derivatives of a multivariate normal cumulative distribution function (MVN CDF) w.r.t. certain elements by using the derivatives of the same MVN CDF w.r.t. ...
- 31
3
votes
1
answer
55
views
Which is the denominator of the Brier score for joint multiple variables predictions?
Brier score can be computed for joint predictions of multiple variables, each with multiple categories.
Let's say we have 4 variables with 3 possible classes each.
In that case, the denominator of the ...
- 403
6
votes
3
answers
194
views
Paradox of Brier skill score of perfectly calibrated output?
Given outcomes, $y \in \{0,1\}$ and outputs $o = f(x) \in \mathbb R, o \in [0,1]$, I'm interested in the case where the model $f$ perfectly models the variable $Y$.
Since $Y$ is Bernoulli, this means
$...
- 19.4k
5
votes
1
answer
58
views
Can we get probabilistic predictions evaluable by proper scoring rules from bayesian inference without evaluating the marginal likelihood?
Let's say we have a vector of inputs, $X=[x_0,\dots, x_{n-1}]$, and a vector of outputs, $Y=[y_0, \dots, y_{n-1}]$.
We would like to predict the distribution of a new output ,$\hat{y}$, given a new ...
- 451
1
vote
0
answers
120
views
Calibrating CatBoostClassifier produces worse results
I'm performing multiclass probability prediction using CatBoostClassifier on a dataset with ~4000 rows, 13 features, 4 target classes. Dataset has outliers, but it is balanced.
For this task I'm using ...
- 23
3
votes
1
answer
85
views
Two-sided KS-Test for Evaluating Prediction Model?
In the article
https://ginimachine.com/blog/machine-learning-model-evaluation/ there is a proposal of using Two-Sided KS-Tests for evaluating the accuracy of predictions from Machine Learning (ML) ...
- 133
0
votes
2
answers
136
views
Why do we maximize likelihood (sum of logs) and not simply maximize sum of probabilities? [duplicate]
In logistic regression we find the maximum likelihood estimator - $\max \prod_{i} p(y_i \mid x_i)$. Which in practice means maximizing the sum of log likelihoods. This makes sense, I understand MLE.
...
11
votes
1
answer
310
views
Reconciling optimisation for log-likelihood and Brier score
Both log-likelihood and Brier score are proper scoring rules. As such, they reach the optimum when the predicted probabilities match the true ones. Since there is only one true probability for each ...
- 9,468
3
votes
2
answers
121
views
What is a scoring rule for binary classification that is not dependent on the "difficulty" of classification?
Consider a model that predicts the probability of some binary event $Y$ (potentially given some features $X$). Denote the estimated probability of $Y$ occurring as $\hat{p}$. One possible choice for a ...
- 441
2
votes
0
answers
72
views
Error metric for regression of count data: Poisson Deviance or Mean Square Error?
I would like to understand what difference it makes, if I use, for example, either Mean Square Error or Poisson Deviance as error metric/loss function for a regression of count data. Are there any a-...
- 2,738
3
votes
1
answer
358
views
How to rank data based on multiple variables
I need help in ranking data, says car models in this case, based on multiple variables. For some variables (eg. mpg), the higher the better. For some variables (eg. car age), the lower the better. For ...
- 79
0
votes
0
answers
53
views
Can the Brier score and Concordance index be anti-correlated?
I am using a proportional hazards Cox model to predict the survival probability of some mechanical components. I am using a combined L1-L2 penalization, and I want to optimize the (integrated) Brier ...
6
votes
1
answer
93
views
Multiclass proper scoring rule decomposition: (weighted) average across the categories?
I have found a Python function that calculates the decomposition of various proper scoring rule, such as Brier score and log loss. However, it does not seem to accept arrays as arguments, so if I want ...
- 65.8k
0
votes
0
answers
23
views
How to show that the influence function of minimum density power divergence estimator with positive tuning parameter is bounded?
In the linked paper, in the influence function section, the term ${u_{\theta}(y)}{f_{\theta}(y)}^\alpha$ is directly called bounded which i do not get the explanation of? Here $\alpha > 0$ is the ...
8
votes
2
answers
426
views
As Brier Score = MSE, does MSE in a regression have a calibration-discrimination decomposition?
When the outcome of a supervised learning problem is binary and probabilities are predicted, Brier score can be decomposed into a measure of calibration and a measure of discrimination.
...
- 65.8k
3
votes
0
answers
124
views
Is Brier score strictly proper in multi-label problems?
In problems where one of $3+$ categories can be observed and we prodict the probability of each category being observed, it is known that the Brier score is a strictly proper scoring rule that is ...
- 65.8k
4
votes
1
answer
98
views
Why isn't there a square root version of the Brier score similar to how RMSE complements MSE?
When computing the mean squared error of a regression model, we get a metric in square units. For ease of interpretation, we can therefore instead compute the root mean squared error, which are in ...
- 107
1
vote
1
answer
30
views
Is it preferred to evaluate with a metric at a single decision threshold (eg Fbeta) vs averageing across thresholds (eg ROC-AUC)
Consider these two approaches to evaluating a classifiers performance:
Choose a metric that summarizes the confusion matrix at a pre-determined decision threshold. Common suggestions seems to be ...
- 107
3
votes
1
answer
163
views
What is the relationship between the Brier score "refinement" and the area under the ROC curve?
In the Wikipedia article on Brier score, there is a claim that the "refinement" in the two-component decomposition of Brier score is related to the area under the receiver-operator ...
- 65.8k
5
votes
1
answer
287
views
Unpack the notation used in Wikipedia's decomposition of the Brier score
Wikipedia has an article about the Brier score whose notation confuses me.
The article starts out easy enough by defining the Brier score to be:
$$
BS = \dfrac{1}{N}\overset{N}{\underset{i = 1}{\sum}}\...
- 65.8k
4
votes
0
answers
100
views
Can the calibration-discrimination decomposition of Brier score be viewed as the bias-variance decomposition of mean squared error?
The mean squared error has a famous decomposition into bias and variance.
$$
\text{MSE} = \text{bias}^2 + \text{var}
$$
Brier score is also a mean squared error calculation, and Brier score has a ...
- 65.8k
3
votes
2
answers
121
views
Do I need to calibrate a model to use the Brier score?
When I use the Brier score loss, do I need to calibrate the model and then use the calibrated model's predictions as input into the Brier score loss?
If I just use a non-calibrated model's ...
- 1,399
10
votes
2
answers
917
views
Why use a scoring rule different from the loss function?
I guess my question is related to these ones: Choosing among proper scoring rules, The performance metric used in prediction is different from the objective function to train the model, but I'm still ...
- 9,468
0
votes
1
answer
70
views
Between steps for fisher information matrix element using Poisson regression?
I am currently working through some math related to my work, and trying to understand how the individual pieces of the following equations come together for the Fisher information matrix expression (...
- 3
2
votes
1
answer
288
views
Model evaluation metrics for comparing predicted probability accuracy across different datasets?
I'm working on an online model scoring framework, my goal is to be able to understand if my model's predictive performance is degrading week-over-week. I have a classification model (trained on binary ...
- 31
0
votes
0
answers
27
views
Best way to show one Bayesian model is more certain and accurate than another, based on simulated data?
I'm trying to compare performance of two bayesian models $A$ and $B$ on simulated data. It's a recruitment curve fitting problem and I'm interested in how accurate these models are in estimating only ...
- 211
2
votes
0
answers
137
views
When *is* classification accuracy the right measure of performance
Plenty has been discussed on Cross Validated about the drawbacks of classification accuracy when it comes to evaluating classification models. One good answer is here, for instance.
Under what ...
- 65.8k
0
votes
1
answer
50
views
Scaled median shift between two observation when median is close to zero
I'm coming for a computer science background and statistics is not my forte, please bear with me.
I have two revisions $R_1$ and $R_2$ each consisting of around 10000 processes $T_i$ (involving some ...
- 101
0
votes
0
answers
19
views
Was approaching this as a classification problem a mistake and should I have to use regression instead?
So I am training a model to predict baseball plate appearance outcomes, which I have been modelling as a single multi-class output problem, namely because single, mutually exclusive outcomes is what ...
2
votes
2
answers
94
views
Ideal scoring rules for multitask classification?
I am seeking advice for the best way to score a multi-output/multitask classification model's output.
Problem setup
A simplified version of the model is as follows:
Training data have F features, say ...
- 23
1
vote
0
answers
44
views
Is generation/evaluation of probabilistic predictions on continuous data feasible for larger data sets in practice?
To better capture uncertainty about the phenomena that we model, probabilistic predictions seem to be a natural and common extension of point predictions.
Methods for evaluation of these predictions ...
- 451
3
votes
1
answer
190
views
Unusual approach to assess a predictive model's performance?
Context: I am working on a predictive model. Let's call it $f$. The outcome that $f$ is trying to predict is binary. It makes predictions as probabilities, i.e. for a given input $x$, $f(x) \in (0,1)$....
- 620
1
vote
0
answers
18
views
Researching the effect of bookmakers' odds on predictions
I did an experiment in which I asked 150 people to predict the likelihood of the home team winning eight upcoming NBA playoff matches. Subjects were separated in four different treatments in a 2x2 ...
- 11
1
vote
2
answers
128
views
First derivative of multivariate normal density with exchangeable correlation structure
As part of a proof, I need to take the first derivative of the log of the following multivariate normal density: $(2\pi)^{-k/2} |\Sigma|^{-1/2} \exp\left(\frac{-1}{2} x'\Sigma^{-1}x\right)$.
In this ...
- 735
1
vote
0
answers
40
views
Is the Wilcoxon Signed Rank Test appropriate when the Brier score is the accuracy metric?
When comparing model performance, is it valid to use the Wilcoxon signed rank test for matched pairs, when the accuracy metric is the Brier score?
(Here, the Brier score is used in calculating the OOB ...
- 363
2
votes
1
answer
161
views
Equivalent of proper scoring rule for point forecasts
Proper scoring rule is a concept used for evaluating density forecasts. What would be an equivalent for evaluating point forecasts? E.g. mean squared error seems like a proper metric for evaluating ...
- 68.8k
1
vote
0
answers
723
views
Creating and interpreting calibration plots for several models with a binary outcome
I have made several models (RF, XGB and GLM) to predict a binary outcome and they all achieved an AUC of approximately 0.8 and Brier scores 0.1-0.15.
Test set is fairly small (n= 350), cases with ...
- 11
1
vote
1
answer
66
views
Suggestions on dealing with outliers when sample size is very small AND you must order the results
I run competitive events. In our normal event, we have 8 adjudicators split between to categories. Skill and Artistry.
For each category we throw out the high and low scores and average the remaining ...
- 11
5
votes
1
answer
1k
views
Is Brier Score appropriate when comparing different classification models?
TL;DR: I am working with binary classifications. I have different models I want to compare their performance out of the box. I read that accuracy is a poor metric, and Brier score or log loss should ...
- 65
2
votes
1
answer
58
views
Practically implementing scoring rules
I am intrigued by the discussion of scoring-rules yet I am left wondering about its practical implementation; I hope this thread can ameliorate that for me and ideally others. Tabling the issue of the ...
- 369
8
votes
2
answers
444
views
Calculating the Brier or log score from the confusion matrix, or from accuracy, sensitivity, specificity, F1 score etc
Suppose I have a confusion matrix, or alternatively any one or more of accuracy, sensitivity, specificity, recall, F1 score or friends for a binary classification problem.
How can I calculate the ...
- 129k
14
votes
2
answers
2k
views
How does the Brier Score break down to (Reliability - Resolution + Uncertainty)?
The Wikipedia page states this in the decompositions section, and it is also stated in an older paper
I have never been able to understand these explanations, and I wonder what I am missing and if ...
- 952
5
votes
1
answer
261
views
Does logistic regression try to predict the true conditional P(Y|X)?
Consider a binary classification dataset (X, Y), generated according to some unknown distribution $P(X, Y)$. I have a question about models which output probabilities by minimizing the cross-entropy ...
- 1,257
1
vote
1
answer
76
views
How to evaluate luck vs skill in judgment accuracy and how to compare different measures of accuracy?
I have data about performance on two types of judgment task (for people), each type with a different format of ground truth for the targets (also people). All judges evaluated all targets, there were ...
- 23
2
votes
0
answers
71
views
$F_1$ score generalised to probabilities: Why squares in the denominator?
I recently stumbled over a generalisation of $F_1$ score to cases where the model predicts probabilities:
$$
F_1 = 2 \frac{\sum y_i \hat{p}_i}{\sum y_i^2 + \sum \hat{p}_i^2}
$$
where $y_i \in \{ 0, 1 \...
- 9,468