Questions tagged [scoring-rules]
Scoring rules are used to assess the accuracy of predicted probabilities, or more generally of predictive densities. Examples of scoring rules include the logarithmic, Brier, spherical, ranked probability and the Dawid-Sebastiani score and the predictive deviance.
261
questions
1
vote
0
answers
66
views
Calibrating CatBoostClassifier produces worse results
I'm performing multiclass probability prediction using CatBoostClassifier on a dataset with ~4000 rows, 13 features, 4 target classes. Dataset has outliers, but it is balanced.
For this task I'm using ...
- 23
0
votes
2
answers
111
views
Why do we maximize likelihood (sum of logs) and not simply maximize sum of probabilities? [duplicate]
In logistic regression we find the maximum likelihood estimator - $\max \prod_{i} p(y_i \mid x_i)$. Which in practice means maximizing the sum of log likelihoods. This makes sense, I understand MLE.
...
- 13.1k
3
votes
1
answer
74
views
Two-sided KS-Test for Evaluating Prediction Model?
In the article
https://ginimachine.com/blog/machine-learning-model-evaluation/ there is a proposal of using Two-Sided KS-Tests for evaluating the accuracy of predictions from Machine Learning (ML) ...
- 124k
3
votes
2
answers
91
views
What is a scoring rule for binary classification that is not dependent on the "difficulty" of classification?
Consider a model that predicts the probability of some binary event $Y$ (potentially given some features $X$). Denote the estimated probability of $Y$ occurring as $\hat{p}$. One possible choice for a ...
- 1,013
10
votes
1
answer
251
views
Reconciling optimisation for log-likelihood and Brier score
Both log-likelihood and Brier score are proper scoring rules. As such, they reach the optimum when the predicted probabilities match the true ones. Since there is only one true probability for each ...
- 92.4k
0
votes
1
answer
250
views
What scoring system to use based on two variables
I have an excel file that shows the performance of several keywords in my paid search campaign.
I have two variables, number of visits & conversion rate, based on which I want to score each ...
CommunityBot
- 1
2
votes
1
answer
477
views
Score/validate a regression Neural Network
I wonder what is the best statistical method to validate/score/evaluate a regression Neural Network used to predict probabilities (an example would be using a Regression NN to predict the probability ...
CommunityBot
- 1
2
votes
0
answers
40
views
Error metric for regression of count data: Poisson Deviance or Mean Square Error?
I would like to understand what difference it makes, if I use, for example, either Mean Square Error or Poisson Deviance as error metric/loss function for a regression of count data. Are there any a-...
- 2,686
1
vote
1
answer
91
views
How to rank data based on multiple variables
I need help in ranking data, says car models in this case, based on multiple variables. For some variables (eg. mpg), the higher the better. For some variables (eg. car age), the lower the better. For ...
54
votes
10
answers
79k
views
How to choose between ROC AUC and F1 score?
I recently completed a Kaggle competition in which roc auc score was used as per competition requirement. Before this project, I normally used f1 score as the metric to measure model performance. ...
- 401
23
votes
3
answers
3k
views
(Why) Is absolute loss not a proper scoring rule?
Brier score is a proper scoring rule and is, at least in the binary classification case, square loss.
$$Brier(y,\hat{y}) = \frac{1}{N} \sum_{i=1}^N\big\vert y_i -\hat{y}_i\big\vert^2$$
Apparently this ...
- 2,155
5
votes
1
answer
68
views
Multiclass proper scoring rule decomposition: (weighted) average across the categories?
I have found a Python function that calculates the decomposition of various proper scoring rule, such as Brier score and log loss. However, it does not seem to accept arrays as arguments, so if I want ...
- 1,013
2
votes
2
answers
453
views
Scoring rules for time series data
I have found quite a lot of articles about scoring rules that seem to first work out theorems and proofs for scoring rules in an iid setting, after which they proceed to apply them to some time series ...
- 1,013
5
votes
3
answers
134
views
Quality measure for predictive Highest Density Regions
An alternative to point, interval and density forecasts/predictions would be "predictive highest density regions (pHDRs)", i.e., HDRs for the conditional density of a yet-unknown future ...
- 1,013
1
vote
1
answer
2k
views
SVC doing great on validation & test data but scored very low on predicted data
First of all, this is my first machine learning project after taking Andrew Ng's course, so please bear with me.
I'm working on the most famous dataset, the Titanic data.
First, I split the dataset ...
CommunityBot
- 1
0
votes
0
answers
40
views
Link between Brier score and AUROC?
The wikipedia article about Brier Score mention a two-way decomposition:
$$ BS = CAL + REL $$ (Calibration and Reliability)
with:
$$ REL = \frac{1}{N} \sum_{k=1}^{K} n_k * \hat{o_k} * (1 - \hat{o_k})$$...
- 1,575
0
votes
0
answers
26
views
Can the Brier score and Concordance index be anti-correlated?
I am using a proportional hazards Cox model to predict the survival probability of some mechanical components. I am using a combined L1-L2 penalization, and I want to optimize the (integrated) Brier ...
8
votes
2
answers
245
views
As Brier Score = MSE, does MSE in a regression have a calibration-discrimination decomposition?
When the outcome of a supervised learning problem is binary and probabilities are predicted, Brier score can be decomposed into a measure of calibration and a measure of discrimination.
...
- 92.1k
0
votes
0
answers
21
views
How to show that the influence function of minimum density power divergence estimator with positive tuning parameter is bounded?
In the linked paper, in the influence function section, the term ${u_{\theta}(y)}{f_{\theta}(y)}^\alpha$ is directly called bounded which i do not get the explanation of? Here $\alpha > 0$ is the ...
4
votes
0
answers
85
views
Can the calibration-discrimination decomposition of Brier score be viewed as the bias-variance decomposition of mean squared error?
The mean squared error has a famous decomposition into bias and variance.
$$
\text{MSE} = \text{bias}^2 + \text{var}
$$
Brier score is also a mean squared error calculation, and Brier score has a ...
- 62.5k
4
votes
1
answer
226
views
Unpack the notation used in Wikipedia's decomposition of the Brier score
Wikipedia has an article about the Brier score whose notation confuses me.
The article starts out easy enough by defining the Brier score to be:
$$
BS = \dfrac{1}{N}\overset{N}{\underset{i = 1}{\sum}}\...
- 1,013
4
votes
1
answer
71
views
Why isn't there a square root version of the Brier score similar to how RMSE complements MSE?
When computing the mean squared error of a regression model, we get a metric in square units. For ease of interpretation, we can therefore instead compute the root mean squared error, which are in ...
- 107
3
votes
0
answers
94
views
Is Brier score strictly proper in multi-label problems?
In problems where one of $3+$ categories can be observed and we prodict the probability of each category being observed, it is known that the Brier score is a strictly proper scoring rule that is ...
- 62.5k
1
vote
1
answer
29
views
Is it preferred to evaluate with a metric at a single decision threshold (eg Fbeta) vs averageing across thresholds (eg ROC-AUC)
Consider these two approaches to evaluating a classifiers performance:
Choose a metric that summarizes the confusion matrix at a pre-determined decision threshold. Common suggestions seems to be ...
- 62.5k
16
votes
3
answers
3k
views
How to choose optimal bin width while calibrating probability models?
Background: There are some great questions/answers here on how to calibrate models which predict probabilities of an outcome happening. For example
Brier score, and its decomposition into resolution, ...
- 1,013
1
vote
0
answers
48
views
What is the relationship between the Brier score "refinement" and the area under the ROC curve?
In the Wikipedia article on Brier score, there is a claim that the "refinement" in the two-component decomposition of Brier score is related to the area under the receiver-operator ...
- 62.5k
3
votes
2
answers
93
views
Do I need to calibrate a model to use the Brier score?
When I use the Brier score loss, do I need to calibrate the model and then use the calibrated model's predictions as input into the Brier score loss?
If I just use a non-calibrated model's ...
- 1,013
241
votes
11
answers
132k
views
Why is accuracy not the best measure for assessing classification models?
This is a general question that was asked indirectly multiple times in here, but it lacks a single authoritative answer. It would be great to have a detailed answer to this for the reference.
...
- 26
7
votes
3
answers
2k
views
How much of neural network overconfidence in predictions can be attributed to modelers optimizing threshold-based metrics?
Neural network "classifiers" output probability scores, and when they are optimized via crossentropy loss (common) or another proper scoring rule, they are optimized in expectation by the ...
- 6,668
2
votes
2
answers
785
views
validation accuracy, recall and precision remains constant after 30th epoch
I am using TensorFlow model EfficientNetB0 for transfer learning, but after a number of epochs the validation accuracy, -precision, and -recall remains constant. Is this something I should be worried ...
CommunityBot
- 1
8
votes
2
answers
753
views
Why use a scoring rule different from the loss function?
I guess my question is related to these ones: Choosing among proper scoring rules, The performance metric used in prediction is different from the objective function to train the model, but I'm still ...
- 124k
0
votes
1
answer
50
views
Between steps for fisher information matrix element using Poisson regression?
I am currently working through some math related to my work, and trying to understand how the individual pieces of the following equations come together for the Fisher information matrix expression (...
- 2,580
0
votes
0
answers
27
views
Best way to show one Bayesian model is more certain and accurate than another, based on simulated data?
I'm trying to compare performance of two bayesian models $A$ and $B$ on simulated data. It's a recruitment curve fitting problem and I'm interested in how accurate these models are in estimating only ...
- 211
4
votes
1
answer
231
views
Factorization of Proper Scoring Rules
Suppose that we have a joint probability distribution $P(X_1,X_2,...,X_n)$. Given a sample $x = (x_1,x_2,...,x_n)$, the proper scoring rule log score can be computed as follows:
$$S(P,x) = \log P(x_1,...
- 1,013
2
votes
1
answer
126
views
Equivalent of proper scoring rule for point forecasts
Proper scoring rule is a concept used for evaluating density forecasts. What would be an equivalent for evaluating point forecasts? E.g. mean squared error seems like a proper metric for evaluating ...
- 1,013
4
votes
1
answer
86
views
Is a "decision boundary" incompatible with proper scoring rules?
Having a decision boundary in a binary classification problem tells me that if the point lies on one side of the boundary, classify as $0$; if the point lies on the other side, classify as $1$.
What ...
- 54.5k
29
votes
2
answers
9k
views
Is accuracy an improper scoring rule in a binary classification setting?
I have recently been learning about proper scoring rules for probabilistic classifiers. Several threads on this website have made a point of emphasizing that accuracy is an improper scoring rule and ...
- 1,013
2
votes
0
answers
135
views
When *is* classification accuracy the right measure of performance
Plenty has been discussed on Cross Validated about the drawbacks of classification accuracy when it comes to evaluating classification models. One good answer is here, for instance.
Under what ...
- 62.5k
4
votes
2
answers
1k
views
Scoring predictions of an ordinal variable
I read about using scoring rules to evaluate the performance of predictive models. In the Wikipedia article about the Brier score, it is stated:
The Brier score is appropriate for binary and ...
- 124k
1
vote
1
answer
42
views
Scaled median shift between two observation when median is close to zero
I'm coming for a computer science background and statistics is not my forte, please bear with me.
I have two revisions $R_1$ and $R_2$ each consisting of around 10000 processes $T_i$ (involving some ...
- 4,726
0
votes
0
answers
19
views
Was approaching this as a classification problem a mistake and should I have to use regression instead?
So I am training a model to predict baseball plate appearance outcomes, which I have been modelling as a single multi-class output problem, namely because single, mutually exclusive outcomes is what ...
- 8,292
2
votes
2
answers
83
views
Ideal scoring rules for multitask classification?
I am seeking advice for the best way to score a multi-output/multitask classification model's output.
Problem setup
A simplified version of the model is as follows:
Training data have F features, say ...
- 66
1
vote
0
answers
43
views
Is generation/evaluation of probabilistic predictions on continuous data feasible for larger data sets in practice?
To better capture uncertainty about the phenomena that we model, probabilistic predictions seem to be a natural and common extension of point predictions.
Methods for evaluation of these predictions ...
- 379
1
vote
0
answers
87
views
Sensitivity, Accuracy, AUROC, Gini
I got following chart:
The algorithms have been applied to a dataset where an outcome is pretty rare, it happens 10% of the times (binary, 0- 90%, 1-10%). It is the response whether a client is going ...
- 18.2k
3
votes
1
answer
190
views
Unusual approach to assess a predictive model's performance?
Context: I am working on a predictive model. Let's call it $f$. The outcome that $f$ is trying to predict is binary. It makes predictions as probabilities, i.e. for a given input $x$, $f(x) \in (0,1)$....
- 620
4
votes
1
answer
102
views
Is the H-measure a strictly proper or a proper scoring rule?
The H-measure was proposed by Dr Hand in his article Measuring classifier performance: a coherent alternative to the area under the ROC curve (2009), as a replacement for AUC-ROC. I haven't been able ...
- 3,058
1
vote
0
answers
16
views
Researching the effect of bookmakers' odds on predictions
I did an experiment in which I asked 150 people to predict the likelihood of the home team winning eight upcoming NBA playoff matches. Subjects were separated in four different treatments in a 2x2 ...
- 11
1
vote
2
answers
119
views
First derivative of multivariate normal density with exchangeable correlation structure
As part of a proof, I need to take the first derivative of the log of the following multivariate normal density: $(2\pi)^{-k/2} |\Sigma|^{-1/2} \exp\left(\frac{-1}{2} x'\Sigma^{-1}x\right)$.
In this ...
- 18.8k
1
vote
0
answers
36
views
Is the Wilcoxon Signed Rank Test appropriate when the Brier score is the accuracy metric?
When comparing model performance, is it valid to use the Wilcoxon signed rank test for matched pairs, when the accuracy metric is the Brier score?
(Here, the Brier score is used in calculating the OOB ...
- 363
18
votes
2
answers
10k
views
Multi-label classification - Brier Score or Log Loss?
I'm using scikit package with RandomForestClassifier, trying to predict binary or multi-lable classifications.
I'm looking for a way to estimate the reliability of the model but really can't figure ...
- 219