Skip to main content

Questions tagged [scoring-rules]

Scoring rules are used to assess the accuracy of predicted probabilities, or more generally of predictive densities. Examples of scoring rules include the logarithmic, Brier, spherical, ranked probability and the Dawid-Sebastiani score and the predictive deviance.

Filter by
Sorted by
Tagged with
5 votes
1 answer
334 views

What is the best epoch to evaluate the test images?

I created a training, a validation and a test set for an image classification task. Then, I did training using the training and did evaluation on validation set. So, the next step is to evaluate the ...
cancan's user avatar
  • 53
0 votes
0 answers
22 views

Comparing performance of probabilistic regression models - how to adapt Brier score?

Suppose I have two predictions models, Model 1 and Model 2. I have a dataset containing observations, features and actual outcomes. For each observation, the “outcomes” (i.e. predictions) that the ...
Alex's user avatar
  • 101
6 votes
1 answer
124 views

There are infinitely many proper scoring rules. Are they all equally valid? Or is log loss superior because of its connection to max likelihood?

I'm kind of obsessed with binary loss functions. How to create a (binary) loss function (scoring rule): Create a function, $f: [0,1] \rightarrow \mathbb{R}_{\geq 0} $, that is symmetric about $x=\...
Chinmay The Math Guy's user avatar
0 votes
0 answers
43 views

Derivative of the multivariate normal cumulative distribution function (CDF) with reparameterisation [duplicate]

I would like to learn how to calculate the derivatives of a multivariate normal cumulative distribution function (MVN CDF) w.r.t. certain elements by using the derivatives of the same MVN CDF w.r.t. ...
Kirin G.'s user avatar
3 votes
1 answer
50 views

Which is the denominator of the Brier score for joint multiple variables predictions?

Brier score can be computed for joint predictions of multiple variables, each with multiple categories. Let's say we have 4 variables with 3 possible classes each. In that case, the denominator of the ...
Antonello's user avatar
  • 403
7 votes
3 answers
172 views

Paradox of Brier skill score of perfectly calibrated output?

Given outcomes, $y \in \{0,1\}$ and outputs $o = f(x) \in \mathbb R, o \in [0,1]$, I'm interested in the case where the model $f$ perfectly models the variable $Y$. Since $Y$ is Bernoulli, this means $...
Firebug's user avatar
  • 19.5k
5 votes
1 answer
56 views

Can we get probabilistic predictions evaluable by proper scoring rules from bayesian inference without evaluating the marginal likelihood?

Let's say we have a vector of inputs, $X=[x_0,\dots, x_{n-1}]$, and a vector of outputs, $Y=[y_0, \dots, y_{n-1}]$. We would like to predict the distribution of a new output ,$\hat{y}$, given a new ...
QMath's user avatar
  • 451
1 vote
0 answers
90 views

Calibrating CatBoostClassifier produces worse results

I'm performing multiclass probability prediction using CatBoostClassifier on a dataset with ~4000 rows, 13 features, 4 target classes. Dataset has outliers, but it is balanced. For this task I'm using ...
primadonna's user avatar
3 votes
1 answer
82 views

Two-sided KS-Test for Evaluating Prediction Model?

In the article https://ginimachine.com/blog/machine-learning-model-evaluation/ there is a proposal of using Two-Sided KS-Tests for evaluating the accuracy of predictions from Machine Learning (ML) ...
Alex's user avatar
  • 133
0 votes
2 answers
125 views

Why do we maximize likelihood (sum of logs) and not simply maximize sum of probabilities? [duplicate]

In logistic regression we find the maximum likelihood estimator - $\max \prod_{i} p(y_i \mid x_i)$. Which in practice means maximizing the sum of log likelihoods. This makes sense, I understand MLE. ...
Amnon Attali's user avatar
11 votes
1 answer
284 views

Reconciling optimisation for log-likelihood and Brier score

Both log-likelihood and Brier score are proper scoring rules. As such, they reach the optimum when the predicted probabilities match the true ones. Since there is only one true probability for each ...
Igor F.'s user avatar
  • 9,418
3 votes
2 answers
107 views

What is a scoring rule for binary classification that is not dependent on the "difficulty" of classification?

Consider a model that predicts the probability of some binary event $Y$ (potentially given some features $X$). Denote the estimated probability of $Y$ occurring as $\hat{p}$. One possible choice for a ...
ischmidt20's user avatar
2 votes
0 answers
61 views

Error metric for regression of count data: Poisson Deviance or Mean Square Error?

I would like to understand what difference it makes, if I use, for example, either Mean Square Error or Poisson Deviance as error metric/loss function for a regression of count data. Are there any a-...
g g's user avatar
  • 2,708
3 votes
1 answer
244 views

How to rank data based on multiple variables

I need help in ranking data, says car models in this case, based on multiple variables. For some variables (eg. mpg), the higher the better. For some variables (eg. car age), the lower the better. For ...
zZzZ's user avatar
  • 79
0 votes
0 answers
40 views

Can the Brier score and Concordance index be anti-correlated?

I am using a proportional hazards Cox model to predict the survival probability of some mechanical components. I am using a combined L1-L2 penalization, and I want to optimize the (integrated) Brier ...
Fernando Cucchietti's user avatar
6 votes
1 answer
86 views

Multiclass proper scoring rule decomposition: (weighted) average across the categories?

I have found a Python function that calculates the decomposition of various proper scoring rule, such as Brier score and log loss. However, it does not seem to accept arrays as arguments, so if I want ...
Dave's user avatar
  • 65.1k
0 votes
0 answers
21 views

How to show that the influence function of minimum density power divergence estimator with positive tuning parameter is bounded?

In the linked paper, in the influence function section, the term ${u_{\theta}(y)}{f_{\theta}(y)}^\alpha$ is directly called bounded which i do not get the explanation of? Here $\alpha > 0$ is the ...
Amlan Dey's user avatar
8 votes
2 answers
389 views

As Brier Score = MSE, does MSE in a regression have a calibration-discrimination decomposition?

When the outcome of a supervised learning problem is binary and probabilities are predicted, Brier score can be decomposed into a measure of calibration and a measure of discrimination. ...
Dave's user avatar
  • 65.1k
3 votes
0 answers
104 views

Is Brier score strictly proper in multi-label problems?

In problems where one of $3+$ categories can be observed and we prodict the probability of each category being observed, it is known that the Brier score is a strictly proper scoring rule that is ...
Dave's user avatar
  • 65.1k
4 votes
1 answer
85 views

Why isn't there a square root version of the Brier score similar to how RMSE complements MSE?

When computing the mean squared error of a regression model, we get a metric in square units. For ease of interpretation, we can therefore instead compute the root mean squared error, which are in ...
another_student's user avatar
1 vote
1 answer
30 views

Is it preferred to evaluate with a metric at a single decision threshold (eg Fbeta) vs averageing across thresholds (eg ROC-AUC)

Consider these two approaches to evaluating a classifiers performance: Choose a metric that summarizes the confusion matrix at a pre-determined decision threshold. Common suggestions seems to be ...
another_student's user avatar
3 votes
1 answer
142 views

What is the relationship between the Brier score "refinement" and the area under the ROC curve?

In the Wikipedia article on Brier score, there is a claim that the "refinement" in the two-component decomposition of Brier score is related to the area under the receiver-operator ...
Dave's user avatar
  • 65.1k
5 votes
1 answer
258 views

Unpack the notation used in Wikipedia's decomposition of the Brier score

Wikipedia has an article about the Brier score whose notation confuses me. The article starts out easy enough by defining the Brier score to be: $$ BS = \dfrac{1}{N}\overset{N}{\underset{i = 1}{\sum}}\...
Dave's user avatar
  • 65.1k
4 votes
0 answers
91 views

Can the calibration-discrimination decomposition of Brier score be viewed as the bias-variance decomposition of mean squared error?

The mean squared error has a famous decomposition into bias and variance. $$ \text{MSE} = \text{bias}^2 + \text{var} $$ Brier score is also a mean squared error calculation, and Brier score has a ...
Dave's user avatar
  • 65.1k
3 votes
2 answers
112 views

Do I need to calibrate a model to use the Brier score?

When I use the Brier score loss, do I need to calibrate the model and then use the calibrated model's predictions as input into the Brier score loss? If I just use a non-calibrated model's ...
Jane Wayne's user avatar
  • 1,399
10 votes
2 answers
873 views

Why use a scoring rule different from the loss function?

I guess my question is related to these ones: Choosing among proper scoring rules, The performance metric used in prediction is different from the objective function to train the model, but I'm still ...
Igor F.'s user avatar
  • 9,418
0 votes
1 answer
65 views

Between steps for fisher information matrix element using Poisson regression?

I am currently working through some math related to my work, and trying to understand how the individual pieces of the following equations come together for the Fisher information matrix expression (...
JHS's user avatar
  • 3
0 votes
0 answers
27 views

Best way to show one Bayesian model is more certain and accurate than another, based on simulated data?

I'm trying to compare performance of two bayesian models $A$ and $B$ on simulated data. It's a recruitment curve fitting problem and I'm interested in how accurate these models are in estimating only ...
chesslad's user avatar
  • 211
2 votes
0 answers
136 views

When *is* classification accuracy the right measure of performance

Plenty has been discussed on Cross Validated about the drawbacks of classification accuracy when it comes to evaluating classification models. One good answer is here, for instance. Under what ...
Dave's user avatar
  • 65.1k
1 vote
1 answer
47 views

Scaled median shift between two observation when median is close to zero

I'm coming for a computer science background and statistics is not my forte, please bear with me. I have two revisions $R_1$ and $R_2$ each consisting of around 10000 processes $T_i$ (involving some ...
Pouya's user avatar
  • 111
0 votes
0 answers
19 views

Was approaching this as a classification problem a mistake and should I have to use regression instead?

So I am training a model to predict baseball plate appearance outcomes, which I have been modelling as a single multi-class output problem, namely because single, mutually exclusive outcomes is what ...
SubtleHyperbole's user avatar
2 votes
2 answers
91 views

Ideal scoring rules for multitask classification?

I am seeking advice for the best way to score a multi-output/multitask classification model's output. Problem setup A simplified version of the model is as follows: Training data have F features, say ...
FidusAchates's user avatar
1 vote
0 answers
44 views

Is generation/evaluation of probabilistic predictions on continuous data feasible for larger data sets in practice?

To better capture uncertainty about the phenomena that we model, probabilistic predictions seem to be a natural and common extension of point predictions. Methods for evaluation of these predictions ...
QMath's user avatar
  • 451
3 votes
1 answer
190 views

Unusual approach to assess a predictive model's performance?

Context: I am working on a predictive model. Let's call it $f$. The outcome that $f$ is trying to predict is binary. It makes predictions as probabilities, i.e. for a given input $x$, $f(x) \in (0,1)$....
dwolfeu's user avatar
  • 620
1 vote
0 answers
18 views

Researching the effect of bookmakers' odds on predictions

I did an experiment in which I asked 150 people to predict the likelihood of the home team winning eight upcoming NBA playoff matches. Subjects were separated in four different treatments in a 2x2 ...
Marc's user avatar
  • 11
1 vote
2 answers
127 views

First derivative of multivariate normal density with exchangeable correlation structure

As part of a proof, I need to take the first derivative of the log of the following multivariate normal density: $(2\pi)^{-k/2} |\Sigma|^{-1/2} \exp\left(\frac{-1}{2} x'\Sigma^{-1}x\right)$. In this ...
bob's user avatar
  • 735
1 vote
0 answers
38 views

Is the Wilcoxon Signed Rank Test appropriate when the Brier score is the accuracy metric?

When comparing model performance, is it valid to use the Wilcoxon signed rank test for matched pairs, when the accuracy metric is the Brier score? (Here, the Brier score is used in calculating the OOB ...
Kyle's user avatar
  • 363
2 votes
1 answer
151 views

Equivalent of proper scoring rule for point forecasts

Proper scoring rule is a concept used for evaluating density forecasts. What would be an equivalent for evaluating point forecasts? E.g. mean squared error seems like a proper metric for evaluating ...
Richard Hardy's user avatar
1 vote
0 answers
683 views

Creating and interpreting calibration plots for several models with a binary outcome

I have made several models (RF, XGB and GLM) to predict a binary outcome and they all achieved an AUC of approximately 0.8 and Brier scores 0.1-0.15. Test set is fairly small (n= 350), cases with ...
mmo's user avatar
  • 11
1 vote
1 answer
51 views

Suggestions on dealing with outliers when sample size is very small AND you must order the results

I run competitive events. In our normal event, we have 8 adjudicators split between to categories. Skill and Artistry. For each category we throw out the high and low scores and average the remaining ...
Omar Paloma's user avatar
5 votes
1 answer
1k views

Is Brier Score appropriate when comparing different classification models?

TL;DR: I am working with binary classifications. I have different models I want to compare their performance out of the box. I read that accuracy is a poor metric, and Brier score or log loss should ...
Luiz Gustavo's user avatar
2 votes
1 answer
57 views

Practically implementing scoring rules

I am intrigued by the discussion of scoring-rules yet I am left wondering about its practical implementation; I hope this thread can ameliorate that for me and ideally others. Tabling the issue of the ...
John Stud's user avatar
  • 369
8 votes
2 answers
430 views

Calculating the Brier or log score from the confusion matrix, or from accuracy, sensitivity, specificity, F1 score etc

Suppose I have a confusion matrix, or alternatively any one or more of accuracy, sensitivity, specificity, recall, F1 score or friends for a binary classification problem. How can I calculate the ...
Stephan Kolassa's user avatar
14 votes
2 answers
2k views

How does the Brier Score break down to (Reliability - Resolution + Uncertainty)?

The Wikipedia page states this in the decompositions section, and it is also stated in an older paper I have never been able to understand these explanations, and I wonder what I am missing and if ...
Joff's user avatar
  • 942
5 votes
1 answer
255 views

Does logistic regression try to predict the true conditional P(Y|X)?

Consider a binary classification dataset (X, Y), generated according to some unknown distribution $P(X, Y)$. I have a question about models which output probabilities by minimizing the cross-entropy ...
usual me's user avatar
  • 1,247
1 vote
1 answer
75 views

How to evaluate luck vs skill in judgment accuracy and how to compare different measures of accuracy?

I have data about performance on two types of judgment task (for people), each type with a different format of ground truth for the targets (also people). All judges evaluated all targets, there were ...
guppy's user avatar
  • 23
2 votes
0 answers
71 views

$F_1$ score generalised to probabilities: Why squares in the denominator?

I recently stumbled over a generalisation of $F_1$ score to cases where the model predicts probabilities: $$ F_1 = 2 \frac{\sum y_i \hat{p}_i}{\sum y_i^2 + \sum \hat{p}_i^2} $$ where $y_i \in \{ 0, 1 \...
Igor F.'s user avatar
  • 9,418
1 vote
1 answer
74 views

What function is used to estimate regression coefficients of a CoxPH model via coxph() in R

I am trying to understand the step by step process of how to estimate the beta coefficients in a CoxPH model. I first get the beta estimates using the coxph() function in r for a sample dataset of six ...
fred's user avatar
  • 11
0 votes
0 answers
28 views

Low RMSE, MSE and a good plot of pred and true values but both R^2 and adjusted r^2 are negative. What am I doing wrong?

I'm developing a code to predict how many bikes will arrive at a given station. The issue is that I'm having a hard time interpreting my metrics: On the one hand, it looks like I have a good MSE and a ...
user14738548's user avatar
0 votes
0 answers
129 views

How to separate two normal distributions which are scores of a machine learning model?

Considering I have a machine learning model that predicts the matching score of two entities. The lower the score is the more different the two entities are and vice versa. ...
Zabir Al Nazi's user avatar

1
2 3 4 5 6