Questions tagged [scoring-rules]

Scoring rules are used to assess the accuracy of predicted probabilities, or more generally of predictive densities. Examples of scoring rules include the logarithmic, Brier, spherical, ranked probability and the Dawid-Sebastiani score and the predictive deviance.

Filter by
Sorted by
Tagged with
0
votes
0answers
5 views

Are there constraints for the variance of predicted probability on calibrated models?

I'm sorry if the title is too vague. I'm not really sure of what I ask, this is a somewhat speculative question... The setting is that I'm using XGBoost in a binary classification problem (40% ...
0
votes
0answers
10 views

XGB model (or any other ML model) objective function vs scoring metrics and log transformations of the target label

I spent some time googling and could not find a proper answer for my question, maybe I have some terms confused but here is the question: When fitting a XGB model (or any ML model like Keras ANN or ...
0
votes
0answers
13 views

Why use log predictive score?

I have seen a density forecasting paper using the log predictive score. There are many loss functions, but the authors suggest that the log score is local and proper. I don't understand why this makes ...
1
vote
1answer
26 views

A question about a logistic regression classifier performance (with and without resampling)

I am working on a dataset with 20 independent variables and 41188 instances. The task is a binary classification where the target variable has 36548 number of no's and 4640 of yes's. I have used ...
4
votes
1answer
38 views

Compare two forecasters on Brier score

I wish to compare two forecasters based on their historical performance (i.e. I want to determine who is better and by how much). The issue is that the two forecasters have performed a different ...
0
votes
1answer
16 views

What scoring system to use based on two variables

I have an excel file that shows the performance of several keywords in my paid search campaign. I have two variables, number of visits & conversion rate, based on which I want to score each ...
0
votes
0answers
5 views

if i want well calibrated probabilities but have class imbalance what metric?

i am having some issues on trying to get a correct metric for an imbalanced problem. it is a credit risk problem where i am trying to predict default of a company so i care about probability output. i ...
0
votes
0answers
5 views

Best way to combine disparate cost vectors to a single cost score scalar

Suppose I have a system with four components each of which may occupy a certain state. Suppose that each state a component is in is associated with a cost vector (or scalar) representing the cost of ...
0
votes
0answers
21 views

Interpreting an integrated brier score that is above 0.25

It is known that the Brier score of a perfect predictive model is 0 while the Brier score of a trivial model is 0.25. However, can I make the same interpretation when looking at a model's integrated ...
0
votes
0answers
12 views

Silhouette score average : per cluster or not?

Wikipedia in English says that the average silhouette for k clusters is simply the average of the silhouette on all samples: $$ S(k) = {1\over{n}} \sum_{i=0}^{n-1} s(i) $$ where s(i) is the silhouette ...
1
vote
0answers
8 views

Normalizing average scores resulted from different number of votes

Imagine a contest, with 30 contestants, and 40 judges. Each judge had to rate them from 0-3, and the average score of each participant will be calculated. But not all participants were rated by all of ...
2
votes
1answer
38 views

Naive benchmarks for scoring rules

I am a non-mathematical R programmer who is completely new to the idea of scoring rules. I would like to start using them instead of classification evaluation measures like accuracy and recall, which ...
1
vote
0answers
23 views

Non-mathematical explanation of how to interpret and evaluate scoring rules in R

I am a non-mathematical R programmer who is completely new to the idea of scoring rules. I would like to start using them instead of classification evaluation measures like accuracy and recall, which ...
2
votes
0answers
21 views

mean squared error or brier score?

i have a classification problem using xgboost, i was optimizing on brier score or 'neg_brier_score' in sklearn. however what is the difference between 'neg_brier_score' and '‘neg_mean_squared_error’ ...
0
votes
0answers
16 views

overfitting and brier score

I have a imbalanced classification problem where i want to see if a client is defaulter or non defaulter. What is important to me is the probability of default, and how well calibrated the model is so ...
2
votes
1answer
43 views

How is the Brier score 'more focused' on the positive class?

I am using Brier score as my scoring metric, as opposed to log loss. My reason for doing so was because I read it was more focused on the positive class than log loss. But how is it? https://...
0
votes
0answers
9 views

How to improve a scoring function without relying too much on the newer elements

I have a $f$ function that is supposed to measure the comprehension of the advantages of a product: $$ \begin{align}f(product_i) = \frac{|ClaimedAdvantages_i - PerceivedAdvantages_i|}{...
2
votes
1answer
36 views

Proper Scoring Rule in Optical Character Recognition

Cross Validated likes to promote proper scoring rules in "classification" problems. That is, get accurate probability predictions. Then make the classifications, taking into account the cost ...
11
votes
2answers
499 views

Reference for log-loss (cross-entropy)?

I'm trying to track down the original reference for the logarithmic loss (logarithmic scoring rule, cross-entropy), usually defined as: $$L_{log}=y_{true} \log(p) + (1-y_{true}) \log(1-p)$$ For the ...
1
vote
1answer
142 views

Brier score of calibrated probs is worse than non calibrated probs

The question is related to probability calibration and Brier score I have faced with the following issue. I have Random forest binary classifier and then I apply isotonic regression to calibration of ...
2
votes
1answer
130 views

what is the scoring variable called for aucpr?

i am trying to conduct a grid search for an imbalanced problem however i cannot find the aucpr (area under curve precision recall) scoring metric for gridsearch. e.g. you have 'roc-auc', 'neg-brier-...
4
votes
1answer
66 views

Checking whether Brier score is a strictly proper scoring rule

I want to check whether Brier Score is a strictly proper scoring rule based on some definition I found here. Since the paper is behind a paywall, I provide the definition here: A scoring rule assigns ...
1
vote
0answers
19 views

Difference between cross entropy/log loss and logarithmic scoring rule?

I ran regressions and random forests using log loss as scoring metric, as suggested here and here. I was reading this which was linked in the second reference, and I started doubting: Is log loss/...
1
vote
0answers
23 views

Science practice: Where to introduce approximations?

In my work, I am using an algorithm which relies on estimates of the gradient of the log-posterior at a collection of Monte Carlo samples. Since this gradient is not available in closed form, I must ...
0
votes
0answers
42 views

Please guide me to make a Formula

Suppose, there are 3 subjects in a school named subject A, subject B and subject C. A student is asked 2 questions for each subject. For example, for subject A, each student is asked Now, I want to ...
13
votes
2answers
287 views

Brier Score and extreme class imbalance

Since I've heard about proper scoring rules for binary classification like the Brier score or Log Loss, I am more and more convinced that they are drastically underrepresented in practice in favor of ...
0
votes
0answers
29 views

Averaging Brier score [duplicate]

To score a RandomForestClassifier using GridSearchCV for multiclass classification, I decided to use Brier score. However, I ...
1
vote
0answers
21 views

Brier score: $L_1$ instead of $L_2$ [duplicate]

Assume that we have some count data $x_{1}, \dots, x_{n}$, which take values $\{1, \dots, m\}$ and we have some estimator of the probability mass function, $\hat{\mathbf{p}} = (\hat{p}_{1}, \dots, \...
0
votes
0answers
53 views

Score function of multi-variate normal distribution

There is an existing question about the score function of multivariate-normal distributions (i.e. the gradient of the log-pdf), but it is somewhat specific to the problem of the OP, and I am ...
0
votes
0answers
17 views

Binary probability scoring: Intuition on why a method might perform better in terms of Brier, log loss but worse in terms of Area under ROC/PR curve?

I'm trying to compare two methods. I have surface knowledge about these scorers, so I've noticed that scorers in which method A performs better are both proper scoring rule, while B performs better in ...
0
votes
1answer
38 views

Calculation of event probability from hazard ratios for new patients

I currently read a publication about a prediction model to estimate heart failure specific events, providing hazard ratios for multiple risk factors (see below only the HRs of one state of 3 that are ...
0
votes
0answers
24 views

Total sum of squares decomposition and Brier score

Building on my previous question, we also can use square loss when we do classification problems (probability of class membership, really). When we use square loss in a classification problem, it’s ...
0
votes
0answers
16 views

Scoring function to avoid overfitting in CV when tuning hyperparameters

I want to apply a Bayesian Optimization process to tune the hyperparameters of my model. Let's assume that I use average precision as performance measure for a given set of hyperparameters. I use n-...
5
votes
1answer
153 views

probability calibration and Brier score

Assume that I have a binary classification problem. The outcome from classification I am mostly interested in is the well-calibrated probabilities. The first way to check this is the calibration plot (...
1
vote
1answer
29 views

Quality measure for predictive Highest Density Regions

An alternative to point, interval and density forecasts/predictions would be "predictive highest density regions (pHDRs)", i.e., HDRs for the conditional density of a yet-unknown future ...
1
vote
1answer
104 views

calibration of classifier scores: isotonic regression

I am investigating the isotonic regression approach to calibrate the scores from a classifier. If I understand correctly, we do the following. First, we get the calibration plot (or reliability curve),...
1
vote
1answer
85 views

intuition behind Brier score

Assume that we have some count data $x_{1}, \dots, x_{n}$, which take values $\{1, \dots, m\}$ and we have some estimator of the probability mass function, $\hat{\mathbf{p}} = (\hat{p}_{1}, \dots, \...
1
vote
0answers
18 views

Is a “decision boundary” incompatible with proper scoring rules?

Having a decision boundary in a binary classification problem tells me that if the point lies on one side of the boundary, classify as $0$; if the point lies on the other side, classify as $1$. What ...
0
votes
0answers
21 views

Risk score decile naming convention

I have what would look a very simple question. Someone told me that specifically for a risk score deciles are inverted in names, that is, the first decile is the highest 10% of my score distribution, ...
4
votes
1answer
119 views

How can proper scoring rules optimize the probabilistic prediction compared to improper scoring rules?

I understand the fundamentals in the decision theory about accuracy being an improper scoring rule compared to other proper scoring rules like ...
2
votes
0answers
38 views

simulation of logistic regression sensitivity to prior probability: Brier score vs accuracy

I wrote a simulation in R that compares performance of a logistic regression as I vary the prior probability of the two classes. The gist is that I do a simulation of data that generates nearly ...
2
votes
2answers
44 views

Classification Models giving probabilities at extreme end only

I am building a binary classification model with proportion of 1 is at only 3% and total 70000 data points.I have 5 variables out of which 3 are coming out to be important. I have built model using ...
2
votes
0answers
30 views

Is the H-measure a strictly proper or a proper scoring rule?

The H-measure was proposed by Dr Hand in his article Measuring classifier performance: a coherent alternative to the area under the ROC curve (2009), as a replacement for AUC-ROC. I haven't been able ...
3
votes
1answer
57 views

Are Brier and log-loss proper or strictly proper scoring rules?

(This article nicely explains the difference between proper and proper scoring rules) According to the Wikipedia entry, and Merkle & Steyvers (2013), these are both strictly proper scoring rules. ...
0
votes
0answers
5 views

Is there a metric for the gains curve?

as we do have a ROC AUC metric, I was wondering if there is a specific metric to evaluate the gain curve?
6
votes
2answers
133 views

What exactly does a proper scoring rule want to do?

I will adapt an excellent simulation by our Stéphane Laurent for this question. ...
17
votes
3answers
2k views

(Why) Is absolute loss not a proper scoring rule?

Brier score is a proper scoring rule and is, at least in the binary classification case, square loss. $$Brier(y,\hat{y}) = \frac{1}{N} \sum_{i=1}^N\big\vert y_i -\hat{y}_i\big\vert^2$$ Apparently this ...
1
vote
1answer
41 views

Question on Rao-Cramer Lower Bound

A question with a solution that I don't quite get: asking for the Cramér-Rao lower bound of a random Poisson sample. If we take the log of the function $f(x; \theta)$ and take its first derivative ...
0
votes
0answers
16 views

Genetic risk score

I performed a GWAS study finding some significant SNPs.Then, through the validation with an external cohort I did not confirm my results (the SNPs found in the previous cohort were not significant in ...
2
votes
1answer
143 views

MLE Asymptotic Normality regularity conditions

I had this lecture of mathematical statistics about asymptotic normality of MLE. In order to prove this, a series of regularity conditions were stated, and the identifiability condition was among them....