Questions tagged [scoring-rules]
Scoring rules are used to assess the accuracy of predicted probabilities, or more generally of predictive densities. Examples of scoring rules include the logarithmic, Brier, spherical, ranked probability and the Dawid-Sebastiani score and the predictive deviance.
234
questions
5
votes
2
answers
501
views
Why use a scoring rule different from the loss function?
I guess my question is related to these ones: Choosing among proper scoring rules, The performance metric used in prediction is different from the objective function to train the model, but I'm still ...
- 8,488
0
votes
1
answer
30
views
Between steps for fisher information matrix element using Poisson regression?
I am currently working through some math related to my work, and trying to understand how the individual pieces of the following equations come together for the Fisher information matrix expression (...
- 3
0
votes
0
answers
23
views
Best way to show one Bayesian model is more certain and accurate than another, based on simulated data?
I'm trying to compare performance of two bayesian models $A$ and $B$ on simulated data. It's a recruitment curve fitting problem and I'm interested in how accurate these models are in estimating only ...
- 211
2
votes
0
answers
88
views
When *is* classification accuracy the right measure of performance
Plenty has been discussed on Cross Validated about the drawbacks of classification accuracy when it comes to evaluating classification models. One good answer is here, for instance.
Under what ...
- 55.2k
1
vote
1
answer
35
views
Scaled median shift between two observation when median is close to zero
I'm coming for a computer science background and statistics is not my forte, please bear with me.
I have two revisions $R_1$ and $R_2$ each consisting of around 10000 processes $T_i$ (involving some ...
- 111
0
votes
0
answers
17
views
Was approaching this as a classification problem a mistake and should I have to use regression instead?
So I am training a model to predict baseball plate appearance outcomes, which I have been modelling as a single multi-class output problem, namely because single, mutually exclusive outcomes is what ...
2
votes
2
answers
49
views
Ideal scoring rules for multitask classification?
I am seeking advice for the best way to score a multi-output/multitask classification model's output.
Problem setup
A simplified version of the model is as follows:
Training data have F features, say ...
- 23
1
vote
0
answers
33
views
Is generation/evaluation of probabilistic predictions on continuous data feasible for larger data sets in practice?
To better capture uncertainty about the phenomena that we model, probabilistic predictions seem to be a natural and common extension of point predictions.
Methods for evaluation of these predictions ...
- 185
3
votes
1
answer
185
views
Unusual approach to assess a predictive model's performance?
Context: I am working on a predictive model. Let's call it $f$. The outcome that $f$ is trying to predict is binary. It makes predictions as probabilities, i.e. for a given input $x$, $f(x) \in (0,1)$....
- 580
1
vote
0
answers
14
views
Researching the effect of bookmakers' odds on predictions
I did an experiment in which I asked 150 people to predict the likelihood of the home team winning eight upcoming NBA playoff matches. Subjects were separated in four different treatments in a 2x2 ...
- 11
1
vote
2
answers
97
views
First derivative of multivariate normal density with exchangeable correlation structure
As part of a proof, I need to take the first derivative of the log of the following multivariate normal density: $(2\pi)^{-k/2} |\Sigma|^{-1/2} \exp\left(\frac{-1}{2} x'\Sigma^{-1}x\right)$.
In this ...
- 714
1
vote
0
answers
28
views
Is the Wilcoxon Signed Rank Test appropriate when the Brier score is the accuracy metric?
When comparing model performance, is it valid to use the Wilcoxon signed rank test for matched pairs, when the accuracy metric is the Brier score?
(Here, the Brier score is used in calculating the OOB ...
- 363
1
vote
1
answer
83
views
Equivalent of proper scoring rule for point forecasts
Proper scoring rule is a concept used for evaluating density forecasts. What would be an equivalent for evaluating point forecasts? E.g. mean squared error seems like a proper metric for evaluating ...
- 63.7k
0
votes
0
answers
45
views
At what point during model development can model calibration be applied?
I have been working on prediction models in R studio based on a rather small data set. There is a total of ~ 1200 cases with 150 to 400 positive cases depending on which of the different outcomes is ...
- 11
1
vote
0
answers
260
views
Creating and interpreting calibration plots for several models with a binary outcome
I have made several models (RF, XGB and GLM) to predict a binary outcome and they all achieved an AUC of approximately 0.8 and Brier scores 0.1-0.15.
Test set is fairly small (n= 350), cases with ...
- 11
1
vote
1
answer
22
views
Suggestions on dealing with outliers when sample size is very small AND you must order the results
I run competitive events. In our normal event, we have 8 adjudicators split between to categories. Skill and Artistry.
For each category we throw out the high and low scores and average the remaining ...
- 11
5
votes
1
answer
386
views
Is Brier Score appropriate when comparing different classification models?
TL;DR: I am working with binary classifications. I have different models I want to compare their performance out of the box. I read that accuracy is a poor metric, and Brier score or log loss should ...
- 65
0
votes
1
answer
43
views
Practically implementing scoring rules
I am intrigued by the discussion of scoring-rules yet I am left wondering about its practical implementation; I hope this thread can ameliorate that for me and ideally others. Tabling the issue of the ...
- 289
5
votes
2
answers
276
views
Calculating the Brier or log score from the confusion matrix, or from accuracy, sensitivity, specificity, F1 score etc
Suppose I have a confusion matrix, or alternatively any one or more of accuracy, sensitivity, specificity, recall, F1 score or friends for a binary classification problem.
How can I calculate the ...
- 115k
0
votes
0
answers
52
views
Creating Brier Score loss function for Catboost in R
Catboost allows the use of Brier Score as a metric, but not for use as a loss function in model training.
I'm attempting to implement Brier score as a custom loss function in R, but I'm finding it a ...
- 21
0
votes
0
answers
14
views
In order to use the model on the most recent data, do they have to be cleaned in the same way as the data used to train the model?
I built Machine Learning model based on for example 100 variables which have been previously cleaned.
Then I saved my ML model in pickle.
Now, I would like to use my ML model to score my clients.
And ...
- 141
0
votes
0
answers
48
views
Should I address the imbalance when using CalibratedClassifierCV?
Im using RandomForestClassifier and XGBClassifier with an imbalanced dataset, 1:2 ratio more or less, 1 being the most prevalent class.
My procedure is the following:
Use StratifiedKFold to get ...
0
votes
0
answers
11
views
a question on model comparison when 2 different training techniques are used (dropout and variational inference)
i have a doubt :
considering 2 different neural networks, one trained through the variational inference technique with denseflipout layers and the other through dropout/concrete dropout
In the first ...
- 325
7
votes
2
answers
552
views
How does the Brier Score break down to (Reliability - Resolution + Uncertainty)?
The Wikipedia page states this in the decompositions section, and it is also stated in an older paper
I have never been able to understand these explanations, and I wonder what I am missing and if ...
- 730
5
votes
1
answer
219
views
Does logistic regression try to predict the true conditional P(Y|X)?
Consider a binary classification dataset (X, Y), generated according to some unknown distribution $P(X, Y)$. I have a question about models which output probabilities by minimizing the cross-entropy ...
- 1,157
1
vote
1
answer
73
views
How to evaluate luck vs skill in judgment accuracy and how to compare different measures of accuracy?
I have data about performance on two types of judgment task (for people), each type with a different format of ground truth for the targets (also people). All judges evaluated all targets, there were ...
- 23
1
vote
0
answers
66
views
$F_1$ score generalised to probabilities: Why squares in the denominator?
I recently stumbled over a generalisation of $F_1$ score to cases where the model predicts probabilities:
$$
F_1 = 2 \frac{\sum y_i \hat{p}_i}{\sum y_i^2 + \sum \hat{p}_i^2}
$$
where $y_i \in \{ 0, 1 \...
- 8,488
1
vote
1
answer
60
views
What function is used to estimate regression coefficients of a CoxPH model via coxph() in R
I am trying to understand the step by step process of how to estimate the beta coefficients in a CoxPH model. I first get the beta estimates using the coxph() function in r for a sample dataset of six ...
- 11
0
votes
0
answers
24
views
Low RMSE, MSE and a good plot of pred and true values but both R^2 and adjusted r^2 are negative. What am I doing wrong?
I'm developing a code to predict how many bikes will arrive at a given station.
The issue is that I'm having a hard time interpreting my metrics:
On the one hand, it looks like I have a good MSE and a ...
- 41
0
votes
0
answers
79
views
How to separate two normal distributions which are scores of a machine learning model?
Considering I have a machine learning model that predicts the matching score of two entities. The lower the score is the more different the two entities are and vice versa.
...
- 115
3
votes
0
answers
77
views
Interpreting the output of scoring rules
Assume we have a probabilistic forecast for a continuous variable. Now we want to validate how good our estimate was. For that, we can use various scoring rules (e.g. CRPS, logarithmic score) or if we ...
- 31
1
vote
0
answers
90
views
Scoring Probabilistic Forecasts - Can we infer a standard deviation from the 84.1 quantile prediction?
I am trying to compare forecasts of a series, and have several trained estimators which are deep neural networks with arbitrary architecture.
I'd like to compare the accuracy of their probabilistic ...
0
votes
1
answer
80
views
Scale-Invariant CRPS Alternative
I am currently working on a probabilistic forecasting problem (outputting the full predictive distribution, possibly in the form of samples) and I need to decide on a measure to evaluate the forecasts....
0
votes
1
answer
241
views
Understanding Brier Loss Composition
I built several models and measured the brier loss, calibration loss, and reliability loss with the direct model and a calibrated one. Now I try to interpret the results, but I cannot understand them ...
- 101
2
votes
1
answer
209
views
Does scoring rules really only apply to categorical outcomes?
The wikipedia article on scoring rule says that
It is applicable to tasks in which predictions must assign probabilities to a set of mutually exclusive outcomes or classes. The set of possible ...
- 367
0
votes
0
answers
281
views
What to do with 99% F1 score in binary classification?
I've been handed a binary classification model to look after. The model uses the F1 score for comparison purposes. The challenge is that the F1 score against the test dataset is very high, like 99%, ...
- 115
0
votes
0
answers
222
views
class_weight='balanced' vs high f_beta score for imbalanced logistic regression in sklearn. Please help explain the difference
I have an imbalanced binary classification problem I am trying to solve with the LogisticRegression algorithm in sklearn.
As the data is highly imbalanced I am looking at ways to treat the imbalance ...
1
vote
0
answers
30
views
Which Proper Scoring rule for when? [duplicate]
Hi, I'm quite new to statistics and have been tasked to evaluate if there is a difference in accuracy between 2 subpopulations in a logistic model.
The credit scoring company's model calculates the ...
- 11
5
votes
1
answer
377
views
Why custom evaluation/scoring metric is causing overfitting in (cross) validation?
I am using machine learning to approach a balanced binary classification task.
Some of rows are more important/valuable than others, so getting them right is extra important. Therefore, to accommodate ...
- 1,521
2
votes
0
answers
192
views
What is the scores calculated from the score function in SelectKBest algorithm? Do they return P value, F score, or others?
I am wondering that how the scores in Select K Best be calculated? I applied the SelectKBest with f_regression as score function. According to the documentation of SelectKBest(https://scikit-learn.org/...
- 21
2
votes
1
answer
546
views
Normalize different binary prediction probability thresholds
I am trying to build an ensemble of three binary classifiers: A, B and C. Each one generates probabilities for the positive class. My goal is to generate a single probability for each case from the ...
- 2,024
5
votes
1
answer
575
views
Which link function in binomial regression is better?
Concerning the choice of the link function in binomial regression (e.g. logit versus probit or cauchit), I wonder what the recommended comparison criterion might be. Note that I am not interested in ...
- 4,515
0
votes
0
answers
31
views
Scoring rules for Dirichlet process mixtures
Let's say I do a sample-based fitting of a Dirichlet process model:
$$
\begin{aligned}
X_i &\sim f(x_i\mid \theta_i)\\
\theta_i &\sim G\\
G &\sim \text{DP}(\alpha, G_0)
\end{aligned}
$$
...
- 245
0
votes
0
answers
221
views
Rating system for zero-sum games that has good theoretical properties compared to Elo rating?
The Elo rating system is widely used in two-player zero-sum games/sports like Chess. It is not obvious to me that the Elo rating system is has the "nicest" theoretical properties out of all ...
- 2,759
0
votes
1
answer
23
views
Can I score Pincode on the basis of financial data?
I have a financial dataset on distinct Pincode level, one of the feature is pincode and other are delinquency, cheque bounce, delinquency amount etc. I am looking to build a model where I can score ...
1
vote
0
answers
39
views
Is half the log loss, twice as good?
Lets say I have two different models based on the same dataset.
Model A has a log loss of 0.30 on this dataset.
Model B has a log loss of 0.60 on this dataset.
If our scoring metric is log loss, is ...
- 223
7
votes
2
answers
258
views
definition of scoring function
Score functions are used in the evaluation of probabilistic forecasts. The question: is the score function positive oriented or negative oriented?
For example, in the paper Strictly Proper Scoring ...
- 73
0
votes
1
answer
335
views
Monte Carlo approximation to find expected value of gradient square
I need to to calculate this term:
$
\mathbb{E}\left[S(Y, L,\theta)S(Y,L,\theta)^\prime\right]
$
Where
$
S(Y,L,\theta) =\frac{\partial}{\partial\theta}
l(Y,L,\theta)
$
With $\theta$ = maximum ...
- 35
1
vote
1
answer
104
views
Undersampling approach in different types of study
If I want to use an undersampling approach to construct the machine learning model, I am wondering if there are any criteria to determine how many times I should sample the data from the majority ...
- 13
8
votes
0
answers
2k
views
Fitting a Logistic Regression via Brier Score or Mean Squared Error
Is there a name for a logistic regression model that has been fit using the Brier score (or equivalently the mean-squared error) rather than the cross-entropy?
I realise this isn't maximum-likelihood, ...
- 52.6k