# Questions tagged [scoring-rules]

Scoring rules are used to assess the accuracy of predicted probabilities, or more generally of predictive densities. Examples of scoring rules include the logarithmic, Brier, spherical, ranked probability and the Dawid-Sebastiani score and the predictive deviance.

219 questions
Filter by
Sorted by
Tagged with
22 views

### Scale-Invariant CRPS Alternative

I am currently working on a probabilistic forecasting problem (outputting the full predictive distribution, possibly in the form of samples) and I need to decide on a measure to evaluate the forecasts....
43 views

### Understanding Brier Loss Composition

I built several models and measured the brier loss, calibration loss, and reliability loss with the direct model and a calibrated one. Now I try to interpret the results, but I cannot understand them ...
133 views

### Does scoring rules really only apply to categorical outcomes?

The wikipedia article on scoring rule says that It is applicable to tasks in which predictions must assign probabilities to a set of mutually exclusive outcomes or classes. The set of possible ...
25 views

### What to do with 99% F1 score in binary classification?

I've been handed a binary classification model to look after. The model uses the F1 score for comparison purposes. The challenge is that the F1 score against the test dataset is very high, like 99%, ...
42 views

### class_weight='balanced' vs high f_beta score for imbalanced logistic regression in sklearn. Please help explain the difference

I have an imbalanced binary classification problem I am trying to solve with the LogisticRegression algorithm in sklearn. As the data is highly imbalanced I am looking at ways to treat the imbalance ...
10 views

### Sum of Confidence Factors in expert systems

I have an expert system for a classification task that contains several rules with associated confidence factors (CF) ...
1 vote
23 views

### Which Proper Scoring rule for when? [duplicate]

Hi, I'm quite new to statistics and have been tasked to evaluate if there is a difference in accuracy between 2 subpopulations in a logistic model. The credit scoring company's model calculates the ...
158 views

### Why custom evaluation/scoring metric is causing overfitting in (cross) validation?

I am using machine learning to approach a balanced binary classification task. Some of rows are more important/valuable than others, so getting them right is extra important. Therefore, to accommodate ...
23 views

### Calculating confidence-score value with probability and IoU

I am trying to fuse two detections with the help of a probability model and, to know that both detections belong to the same object, I use the Intersection over Union (IoU) score. So, the confidence-...
65 views

### What is the scores calculated from the score function in SelectKBest algorithm? Do they return P value, F score, or others?

I am wondering that how the scores in Select K Best be calculated? I applied the SelectKBest with f_regression as score function. According to the documentation of SelectKBest(https://scikit-learn.org/...
46 views

### Bounding The Brier Skill Score

Consider models $M^f,M^r$ (focus and reference), which obtain Brier scores $BS^f,BS^r$ over some testing dataset. The Brier skill score is defined as $BSS=1-\frac{BS^f}{BS^r}$. Generally speaking, BSS ...
1 vote
105 views

### Normalize different binary prediction probability thresholds

I am trying to build an ensemble of three binary classifiers: A, B and C. Each one generates probabilities for the positive class. My goal is to generate a single probability for each case from the ...
277 views

### Which link function in binomial regression is better?

Concerning the choice of the link function in binomial regression (e.g. logit versus probit or cauchit), I wonder what the recommended comparison criterion might be. Note that I am not interested in ...
24 views

### Scoring rules for Dirichlet process mixtures

Let's say I do a sample-based fitting of a Dirichlet process model: \begin{aligned} X_i &\sim f(x_i\mid \theta_i)\\ \theta_i &\sim G\\ G &\sim \text{DP}(\alpha, G_0) \end{aligned} ...
61 views

### Use custom test data with GridSearchCV

When searching for parameters with GridSearchCV, I encountered the problem of getting decent scores with my training data, but bad results with my test data. The design of experiments were conducted ...
42 views

### Rating system for zero-sum games that has good theoretical properties compared to Elo rating?

The Elo rating system is widely used in two-player zero-sum games/sports like Chess. It is not obvious to me that the Elo rating system is has the "nicest" theoretical properties out of all ...
22 views

### Can I score Pincode on the basis of financial data?

I have a financial dataset on distinct Pincode level, one of the feature is pincode and other are delinquency, cheque bounce, delinquency amount etc. I am looking to build a model where I can score ...
1 vote
20 views

### Is half the log loss, twice as good?

Lets say I have two different models based on the same dataset. Model A has a log loss of 0.30 on this dataset. Model B has a log loss of 0.60 on this dataset. If our scoring metric is log loss, is ...
15 views

### How should I score probability data to clearly highlight good/bad results, in a way that doesn't converge on zero?

I'm looking at a problem right now where we see probability data across many results, and I want to take this data and provide an eventual score, which we can then use to judge the overall probability/...
214 views

### definition of scoring function

Score functions are used in the evaluation of probabilistic forecasts. The question: is the score function positive oriented or negative oriented? For example, in the paper Strictly Proper Scoring ...
140 views

### Monte Carlo approximation to find expected value of gradient square

I need to to calculate this term: $\mathbb{E}\left[S(Y, L,\theta)S(Y,L,\theta)^\prime\right]$ Where $S(Y,L,\theta) =\frac{\partial}{\partial\theta} l(Y,L,\theta)$ With $\theta$ = maximum ...
1 vote
52 views

### Undersampling approach in different types of study

If I want to use an undersampling approach to construct the machine learning model, I am wondering if there are any criteria to determine how many times I should sample the data from the majority ...
244 views

### Fitting a Logistic Regression via Brier Score or Mean Squared Error

Is there a name for a logistic regression model that has been fit using the Brier score (or equivalently the mean-squared error) rather than the cross-entropy? I realise this isn't maximum-likelihood, ...
94 views

### What is the link between probabilistic predictions and Bayes optimum decisions?

Frank Harrell writes in one of our community wikis about the link between "Bayes optimum decision" and the link to probabilistic predictions (and, thus, one of his favorite topics in proper ...
47 views

### Weighted vs Non weighted scores for unbalanced classes

I have a dataset which has 99.8% negatives and 0.2% positives. The scores I have got for my model(built using XGBoost) is as follows: ...
138 views

### Scoring Rule for Continuous Probability Prediction

I have a question about choosing the right scoring rule. I am building a system which predicts the spatial (2D) probability of an event. The label data contains continuous values between 0 and 1, ...
1 vote
28 views

### Evaluating a true classifier e.g., pregnancy test

Most alleged "classifiers" give probabilities of class membership. One can use a threshold to map those probabilities to discrete categories, but statisticians are in favor of direct ...
1 vote
79 views

### Evaluating a multi-step forecasting model?

The literature is a bit confusing for me on this one, from what I understand, a great deal of papers evaluate multi-step forecasting models on a single forecasting horizon on the hold out set. It ...
479 views

### How much of neural network overconfidence in predictions can be attributed to modelers optimizing threshold-based metrics?

Neural network "classifiers" output probability scores, and when they are optimized via crossentropy loss (common) or another proper scoring rule, they are optimized in expectation by the ...
1 vote
27 views

### Accuracy of ordinal predictions of a continuous outcome (without fixed thresholds)

I have ordinal predictions by experts: 1, 2, 3, 4, 5. The outcome to be predicted is, however, continuous: 0-36. I don't know what threshold the experts use for the categories 1-5, but I know that the ...
325 views

### Is there something like a confusion matrix for a probabilistic score rather than categories?

Imagine we have pictures of three animals: dogs, cats, and horses. We train our image classifier and get a confusion matrix, noticing that the model tends to predict that dogs are horses. But then we ...
1 vote
2k views

### On which set (train/val/test) do people calculate F1 score, precision and recall?

This may be a stupid question, but when I was looking at the definition of precision/recall etc. it was not mentioned anywhere which set (training/validation/test) this metric should be calculated ...
21 views

### Is it harder to spoof (adversarial examples) a model trained to optimize a proper scoring rule than an improper scoring rule?

The way I figure, if we train a model to stick points on the correct side of a threshold like $0.5$, then all we have to do is tweak a $1$ with a predicted probability of $0.51$ to give a predicted ...
1 vote
47 views

### Validation loss decreasing but accuracy increasing - early stop on accuracy?

I'm designing an image classifier for places365. The validation loss after the first epoch is lower than all other epochs (9.65 cross entropy loss, 11.07 after 10th epoch), but equally the validation ...
66 views

### How to aggregate log-likelihood score of many models?

I have a process through which I estimate the parameters of a model in order to make predictions. Through this process, I end up with many models (with different parameter values) that are then used ...
30 views

### Sum of sensitivity and specificity meaningful?

I came across an article in the BMJ that claims that a useful rule of thumb metric for assessing a medical test's performance is that the sum of specificity and sensitivity should be greater than 1.5 ...
92 views

### How to modify a logloss scoring rule for forecasted probabilities being a Beta distribution?

I've been reading up on scoring rules and posts such as this one Why is LogLoss preferred over other proper scoring rules? It's clear to me that if I have a set of biased coin and after examining each ...
251 views

### validation accuracy, recall and precision remains constant after 30th epoch

I am using TensorFlow model EfficientNetB0 for transfer learning, but after a number of epochs the validation accuracy, -precision, and -recall remains constant. Is this something I should be worried ...
1 vote
38 views

### Sensitivity, Accuracy, AUROC, Gini

I got following chart: The algorithms have been applied to a dataset where an outcome is pretty rare, it happens 10% of the times (binary, 0- 90%, 1-10%). It is the response whether a client is going ...
52 views

### In Bayesian modeling, are there any approaches to score the confidence of the predicted distribution, instead of just the point estimate?

In a situation where one is predicting a distribution, rather than just a point estimate, what are some approaches to check the goodness of fit on observed data? The goal here would be to get a metric ...
466 views

### Circular reasoning in Harrell BBR 18-19?

I am looking at chapter 18 (Information Loss) of Harrell's "Biostatistics for Biomedical Reserach": https://hbiostat.org/doc/bbr.pdf. The example of 18-19 seems like circular reasoning. He ...
92 views

### logistic regression: compare performance to human prediction

Say I want to predict whether a patient develops a disorder or not. I have two prediction 'models': Clinicians estimating the probability of a patient developing the disorder and logistic regression ...
1 vote
109 views

### Find covariance of estimator and derivative of the log-likelihood function

Problem: Given and estimator $\hat k$. The estimation method is unknown (so, it can be max. likelihood, method of moments or another method), however, we know that $bias(\hat k) = 0$. Let $L$ be the ...
1 vote
36 views

### Forecast scores, proper scores and strictly proper scores

Let $\mu_F$ and $\sigma_F^2$ be the expectation and variance of a forecast model $F$ for observations $y$, and let $p_F(\cdot)$ denote the probability density function for the forecast model. Also let ...
169 views

### Can I measure the accuracy of a range of quantiles in my forecast distribution?

I am forecasting items and measuring the point forecast and distribution accuracy of numerous different models against actuals. To measure distribution accuracy I am using the continuous-ranked ...
1 vote
102 views

### About computation of Brier score

Assume that we have some count data $x_{1}, \dots, x_{n}$, generated by probability mass function $\textbf{p} = \{p_{1}, \dots, p_{s} \}$. Let $\hat{\theta}$ be some estimator of $\textbf{p}$. In ...
1 vote
190 views

### Intuition behind Brier score weighing step for censored data

Sources seem to suggest that when calculating Brier scores involving right-censored data, one must weigh the otherwise mean square error function with the inverse probability of censoring weights ...
1 vote
425 views

### F1 weighted vs. Log loss in SciKit learn RandomSearchCV

I am sorry to ask another question regarding this topic but I am still puzzled about the following: When I use 'F1_weighted' as my scoring argument in a ...