# Questions tagged [scoring-rules]

Scoring rules are used to assess the accuracy of predicted probabilities, or more generally of predictive densities. Examples of scoring rules include the logarithmic, Brier, spherical, ranked probability and the Dawid-Sebastiani score and the predictive deviance.

187 questions
Filter by
Sorted by
Tagged with
5 views

### Are there constraints for the variance of predicted probability on calibrated models?

I'm sorry if the title is too vague. I'm not really sure of what I ask, this is a somewhat speculative question... The setting is that I'm using XGBoost in a binary classification problem (40% ...
10 views

### XGB model (or any other ML model) objective function vs scoring metrics and log transformations of the target label

I spent some time googling and could not find a proper answer for my question, maybe I have some terms confused but here is the question: When fitting a XGB model (or any ML model like Keras ANN or ...
13 views

### Why use log predictive score?

I have seen a density forecasting paper using the log predictive score. There are many loss functions, but the authors suggest that the log score is local and proper. I don't understand why this makes ...
26 views

### A question about a logistic regression classifier performance (with and without resampling)

I am working on a dataset with 20 independent variables and 41188 instances. The task is a binary classification where the target variable has 36548 number of no's and 4640 of yes's. I have used ...
38 views

### Compare two forecasters on Brier score

I wish to compare two forecasters based on their historical performance (i.e. I want to determine who is better and by how much). The issue is that the two forecasters have performed a different ...
16 views

### What scoring system to use based on two variables

I have an excel file that shows the performance of several keywords in my paid search campaign. I have two variables, number of visits & conversion rate, based on which I want to score each ...
5 views

### if i want well calibrated probabilities but have class imbalance what metric?

i am having some issues on trying to get a correct metric for an imbalanced problem. it is a credit risk problem where i am trying to predict default of a company so i care about probability output. i ...
5 views

### Best way to combine disparate cost vectors to a single cost score scalar

Suppose I have a system with four components each of which may occupy a certain state. Suppose that each state a component is in is associated with a cost vector (or scalar) representing the cost of ...
21 views

### Interpreting an integrated brier score that is above 0.25

It is known that the Brier score of a perfect predictive model is 0 while the Brier score of a trivial model is 0.25. However, can I make the same interpretation when looking at a model's integrated ...
12 views

### Silhouette score average : per cluster or not?

Wikipedia in English says that the average silhouette for k clusters is simply the average of the silhouette on all samples: $$S(k) = {1\over{n}} \sum_{i=0}^{n-1} s(i)$$ where s(i) is the silhouette ...
8 views

### Normalizing average scores resulted from different number of votes

Imagine a contest, with 30 contestants, and 40 judges. Each judge had to rate them from 0-3, and the average score of each participant will be calculated. But not all participants were rated by all of ...
38 views

### Naive benchmarks for scoring rules

I am a non-mathematical R programmer who is completely new to the idea of scoring rules. I would like to start using them instead of classification evaluation measures like accuracy and recall, which ...
23 views

### Non-mathematical explanation of how to interpret and evaluate scoring rules in R

I am a non-mathematical R programmer who is completely new to the idea of scoring rules. I would like to start using them instead of classification evaluation measures like accuracy and recall, which ...
21 views

### mean squared error or brier score?

i have a classification problem using xgboost, i was optimizing on brier score or 'neg_brier_score' in sklearn. however what is the difference between 'neg_brier_score' and '‘neg_mean_squared_error’ ...
16 views

### overfitting and brier score

I have a imbalanced classification problem where i want to see if a client is defaulter or non defaulter. What is important to me is the probability of default, and how well calibrated the model is so ...
43 views

### How is the Brier score 'more focused' on the positive class?

I am using Brier score as my scoring metric, as opposed to log loss. My reason for doing so was because I read it was more focused on the positive class than log loss. But how is it? https://...
9 views

41 views

### Question on Rao-Cramer Lower Bound

A question with a solution that I don't quite get: asking for the Cramér-Rao lower bound of a random Poisson sample. If we take the log of the function $f(x; \theta)$ and take its first derivative ...