# Questions tagged [scoring-rules]

Scoring rules are used to assess the accuracy of predicted probabilities, or more generally of predictive densities. Examples of scoring rules include the logarithmic, Brier, spherical, ranked probability and the Dawid-Sebastiani score and the predictive deviance.

234 questions
Filter by
Sorted by
Tagged with
501 views

### Why use a scoring rule different from the loss function?

I guess my question is related to these ones: Choosing among proper scoring rules, The performance metric used in prediction is different from the objective function to train the model, but I'm still ...
30 views

### Between steps for fisher information matrix element using Poisson regression?

I am currently working through some math related to my work, and trying to understand how the individual pieces of the following equations come together for the Fisher information matrix expression (...
23 views

### Best way to show one Bayesian model is more certain and accurate than another, based on simulated data?

I'm trying to compare performance of two bayesian models $A$ and $B$ on simulated data. It's a recruitment curve fitting problem and I'm interested in how accurate these models are in estimating only ...
88 views

### When *is* classification accuracy the right measure of performance

Plenty has been discussed on Cross Validated about the drawbacks of classification accuracy when it comes to evaluating classification models. One good answer is here, for instance. Under what ...
1 vote
35 views

### Scaled median shift between two observation when median is close to zero

I'm coming for a computer science background and statistics is not my forte, please bear with me. I have two revisions $R_1$ and $R_2$ each consisting of around 10000 processes $T_i$ (involving some ...
17 views

### Was approaching this as a classification problem a mistake and should I have to use regression instead?

So I am training a model to predict baseball plate appearance outcomes, which I have been modelling as a single multi-class output problem, namely because single, mutually exclusive outcomes is what ...
49 views

### Ideal scoring rules for multitask classification?

I am seeking advice for the best way to score a multi-output/multitask classification model's output. Problem setup A simplified version of the model is as follows: Training data have F features, say ...
1 vote
33 views

### Is generation/evaluation of probabilistic predictions on continuous data feasible for larger data sets in practice?

To better capture uncertainty about the phenomena that we model, probabilistic predictions seem to be a natural and common extension of point predictions. Methods for evaluation of these predictions ...
185 views

### Unusual approach to assess a predictive model's performance?

Context: I am working on a predictive model. Let's call it $f$. The outcome that $f$ is trying to predict is binary. It makes predictions as probabilities, i.e. for a given input $x$, $f(x) \in (0,1)$....
1 vote
14 views

### Researching the effect of bookmakers' odds on predictions

I did an experiment in which I asked 150 people to predict the likelihood of the home team winning eight upcoming NBA playoff matches. Subjects were separated in four different treatments in a 2x2 ...
1 vote
97 views

### First derivative of multivariate normal density with exchangeable correlation structure

As part of a proof, I need to take the first derivative of the log of the following multivariate normal density: $(2\pi)^{-k/2} |\Sigma|^{-1/2} \exp\left(\frac{-1}{2} x'\Sigma^{-1}x\right)$. In this ...
1 vote
28 views

### Is the Wilcoxon Signed Rank Test appropriate when the Brier score is the accuracy metric?

When comparing model performance, is it valid to use the Wilcoxon signed rank test for matched pairs, when the accuracy metric is the Brier score? (Here, the Brier score is used in calculating the OOB ...
1 vote
83 views

### Equivalent of proper scoring rule for point forecasts

Proper scoring rule is a concept used for evaluating density forecasts. What would be an equivalent for evaluating point forecasts? E.g. mean squared error seems like a proper metric for evaluating ...
45 views

### At what point during model development can model calibration be applied?

I have been working on prediction models in R studio based on a rather small data set. There is a total of ~ 1200 cases with 150 to 400 positive cases depending on which of the different outcomes is ...
1 vote
260 views

### Creating and interpreting calibration plots for several models with a binary outcome

I have made several models (RF, XGB and GLM) to predict a binary outcome and they all achieved an AUC of approximately 0.8 and Brier scores 0.1-0.15. Test set is fairly small (n= 350), cases with ...
1 vote
22 views

### Suggestions on dealing with outliers when sample size is very small AND you must order the results

I run competitive events. In our normal event, we have 8 adjudicators split between to categories. Skill and Artistry. For each category we throw out the high and low scores and average the remaining ...
386 views

### Is Brier Score appropriate when comparing different classification models?

TL;DR: I am working with binary classifications. I have different models I want to compare their performance out of the box. I read that accuracy is a poor metric, and Brier score or log loss should ...
43 views

### Practically implementing scoring rules

I am intrigued by the discussion of scoring-rules yet I am left wondering about its practical implementation; I hope this thread can ameliorate that for me and ideally others. Tabling the issue of the ...
276 views

### Calculating the Brier or log score from the confusion matrix, or from accuracy, sensitivity, specificity, F1 score etc

Suppose I have a confusion matrix, or alternatively any one or more of accuracy, sensitivity, specificity, recall, F1 score or friends for a binary classification problem. How can I calculate the ...
52 views

### Creating Brier Score loss function for Catboost in R

Catboost allows the use of Brier Score as a metric, but not for use as a loss function in model training. I'm attempting to implement Brier score as a custom loss function in R, but I'm finding it a ...
14 views

### In order to use the model on the most recent data, do they have to be cleaned in the same way as the data used to train the model?

I built Machine Learning model based on for example 100 variables which have been previously cleaned. Then I saved my ML model in pickle. Now, I would like to use my ML model to score my clients. And ...
48 views

### Should I address the imbalance when using CalibratedClassifierCV?

Im using RandomForestClassifier and XGBClassifier with an imbalanced dataset, 1:2 ratio more or less, 1 being the most prevalent class. My procedure is the following: Use StratifiedKFold to get ...
11 views

### a question on model comparison when 2 different training techniques are used (dropout and variational inference)

i have a doubt : considering 2 different neural networks, one trained through the variational inference technique with denseflipout layers and the other through dropout/concrete dropout In the first ...
552 views

### How does the Brier Score break down to (Reliability - Resolution + Uncertainty)?

The Wikipedia page states this in the decompositions section, and it is also stated in an older paper I have never been able to understand these explanations, and I wonder what I am missing and if ...
219 views

### Does logistic regression try to predict the true conditional P(Y|X)?

Consider a binary classification dataset (X, Y), generated according to some unknown distribution $P(X, Y)$. I have a question about models which output probabilities by minimizing the cross-entropy ...
1 vote
73 views

### How to evaluate luck vs skill in judgment accuracy and how to compare different measures of accuracy?

I have data about performance on two types of judgment task (for people), each type with a different format of ground truth for the targets (also people). All judges evaluated all targets, there were ...
1 vote
66 views

1 vote