Questions tagged [scoring-rules]

Scoring rules are used to assess the accuracy of predicted probabilities, or more generally of predictive densities. Examples of scoring rules include the logarithmic, Brier, spherical, ranked probability and the Dawid-Sebastiani score and the predictive deviance.

Filter by
Sorted by
Tagged with
0 votes
1 answer
22 views

Scale-Invariant CRPS Alternative

I am currently working on a probabilistic forecasting problem (outputting the full predictive distribution, possibly in the form of samples) and I need to decide on a measure to evaluate the forecasts....
user avatar
0 votes
1 answer
43 views

Understanding Brier Loss Composition

I built several models and measured the brier loss, calibration loss, and reliability loss with the direct model and a calibrated one. Now I try to interpret the results, but I cannot understand them ...
user avatar
  • 101
2 votes
1 answer
133 views

Does scoring rules really only apply to categorical outcomes?

The wikipedia article on scoring rule says that It is applicable to tasks in which predictions must assign probabilities to a set of mutually exclusive outcomes or classes. The set of possible ...
user avatar
0 votes
0 answers
25 views

What to do with 99% F1 score in binary classification?

I've been handed a binary classification model to look after. The model uses the F1 score for comparison purposes. The challenge is that the F1 score against the test dataset is very high, like 99%, ...
user avatar
  • 111
0 votes
0 answers
42 views

class_weight='balanced' vs high f_beta score for imbalanced logistic regression in sklearn. Please help explain the difference

I have an imbalanced binary classification problem I am trying to solve with the LogisticRegression algorithm in sklearn. As the data is highly imbalanced I am looking at ways to treat the imbalance ...
user avatar
0 votes
0 answers
10 views

Sum of Confidence Factors in expert systems

I have an expert system for a classification task that contains several rules with associated confidence factors (CF) ...
user avatar
  • 1,119
1 vote
0 answers
23 views

Which Proper Scoring rule for when? [duplicate]

Hi, I'm quite new to statistics and have been tasked to evaluate if there is a difference in accuracy between 2 subpopulations in a logistic model. The credit scoring company's model calculates the ...
user avatar
4 votes
1 answer
158 views

Why custom evaluation/scoring metric is causing overfitting in (cross) validation?

I am using machine learning to approach a balanced binary classification task. Some of rows are more important/valuable than others, so getting them right is extra important. Therefore, to accommodate ...
user avatar
0 votes
0 answers
23 views

Calculating confidence-score value with probability and IoU

I am trying to fuse two detections with the help of a probability model and, to know that both detections belong to the same object, I use the Intersection over Union (IoU) score. So, the confidence-...
user avatar
2 votes
0 answers
65 views

What is the scores calculated from the score function in SelectKBest algorithm? Do they return P value, F score, or others?

I am wondering that how the scores in Select K Best be calculated? I applied the SelectKBest with f_regression as score function. According to the documentation of SelectKBest(https://scikit-learn.org/...
user avatar
  • 21
0 votes
0 answers
46 views

Bounding The Brier Skill Score

Consider models $M^f,M^r$ (focus and reference), which obtain Brier scores $BS^f,BS^r$ over some testing dataset. The Brier skill score is defined as $BSS=1-\frac{BS^f}{BS^r}$. Generally speaking, BSS ...
user avatar
  • 2,381
1 vote
1 answer
105 views

Normalize different binary prediction probability thresholds

I am trying to build an ensemble of three binary classifiers: A, B and C. Each one generates probabilities for the positive class. My goal is to generate a single probability for each case from the ...
user avatar
  • 1,627
5 votes
1 answer
277 views

Which link function in binomial regression is better?

Concerning the choice of the link function in binomial regression (e.g. logit versus probit or cauchit), I wonder what the recommended comparison criterion might be. Note that I am not interested in ...
user avatar
  • 3,111
0 votes
0 answers
24 views

Scoring rules for Dirichlet process mixtures

Let's say I do a sample-based fitting of a Dirichlet process model: $$ \begin{aligned} X_i &\sim f(x_i\mid \theta_i)\\ \theta_i &\sim G\\ G &\sim \text{DP}(\alpha, G_0) \end{aligned} $$ ...
user avatar
  • 193
0 votes
0 answers
61 views

Use custom test data with GridSearchCV

When searching for parameters with GridSearchCV, I encountered the problem of getting decent scores with my training data, but bad results with my test data. The design of experiments were conducted ...
user avatar
  • 101
0 votes
0 answers
42 views

Rating system for zero-sum games that has good theoretical properties compared to Elo rating?

The Elo rating system is widely used in two-player zero-sum games/sports like Chess. It is not obvious to me that the Elo rating system is has the "nicest" theoretical properties out of all ...
user avatar
  • 2,179
0 votes
1 answer
22 views

Can I score Pincode on the basis of financial data?

I have a financial dataset on distinct Pincode level, one of the feature is pincode and other are delinquency, cheque bounce, delinquency amount etc. I am looking to build a model where I can score ...
user avatar
1 vote
0 answers
20 views

Is half the log loss, twice as good?

Lets say I have two different models based on the same dataset. Model A has a log loss of 0.30 on this dataset. Model B has a log loss of 0.60 on this dataset. If our scoring metric is log loss, is ...
user avatar
0 votes
0 answers
15 views

How should I score probability data to clearly highlight good/bad results, in a way that doesn't converge on zero?

I'm looking at a problem right now where we see probability data across many results, and I want to take this data and provide an eventual score, which we can then use to judge the overall probability/...
user avatar
7 votes
2 answers
214 views

definition of scoring function

Score functions are used in the evaluation of probabilistic forecasts. The question: is the score function positive oriented or negative oriented? For example, in the paper Strictly Proper Scoring ...
user avatar
  • 73
0 votes
1 answer
140 views

Monte Carlo approximation to find expected value of gradient square

I need to to calculate this term: $ \mathbb{E}\left[S(Y, L,\theta)S(Y,L,\theta)^\prime\right] $ Where $ S(Y,L,\theta) =\frac{\partial}{\partial\theta} l(Y,L,\theta) $ With $\theta$ = maximum ...
user avatar
1 vote
1 answer
52 views

Undersampling approach in different types of study

If I want to use an undersampling approach to construct the machine learning model, I am wondering if there are any criteria to determine how many times I should sample the data from the majority ...
user avatar
5 votes
0 answers
244 views

Fitting a Logistic Regression via Brier Score or Mean Squared Error

Is there a name for a logistic regression model that has been fit using the Brier score (or equivalently the mean-squared error) rather than the cross-entropy? I realise this isn't maximum-likelihood, ...
user avatar
3 votes
1 answer
94 views

What is the link between probabilistic predictions and Bayes optimum decisions?

Frank Harrell writes in one of our community wikis about the link between "Bayes optimum decision" and the link to probabilistic predictions (and, thus, one of his favorite topics in proper ...
user avatar
  • 32.4k
3 votes
0 answers
47 views

Weighted vs Non weighted scores for unbalanced classes

I have a dataset which has 99.8% negatives and 0.2% positives. The scores I have got for my model(built using XGBoost) is as follows: ...
user avatar
3 votes
1 answer
138 views

Scoring Rule for Continuous Probability Prediction

I have a question about choosing the right scoring rule. I am building a system which predicts the spatial (2D) probability of an event. The label data contains continuous values between 0 and 1, ...
user avatar
1 vote
0 answers
28 views

Evaluating a true classifier e.g., pregnancy test

Most alleged "classifiers" give probabilities of class membership. One can use a threshold to map those probabilities to discrete categories, but statisticians are in favor of direct ...
user avatar
  • 32.4k
1 vote
1 answer
79 views

Evaluating a multi-step forecasting model?

The literature is a bit confusing for me on this one, from what I understand, a great deal of papers evaluate multi-step forecasting models on a single forecasting horizon on the hold out set. It ...
user avatar
  • 123
7 votes
2 answers
479 views

How much of neural network overconfidence in predictions can be attributed to modelers optimizing threshold-based metrics?

Neural network "classifiers" output probability scores, and when they are optimized via crossentropy loss (common) or another proper scoring rule, they are optimized in expectation by the ...
user avatar
  • 32.4k
1 vote
0 answers
27 views

Accuracy of ordinal predictions of a continuous outcome (without fixed thresholds)

I have ordinal predictions by experts: 1, 2, 3, 4, 5. The outcome to be predicted is, however, continuous: 0-36. I don't know what threshold the experts use for the categories 1-5, but I know that the ...
user avatar
  • 111
7 votes
2 answers
325 views

Is there something like a confusion matrix for a probabilistic score rather than categories?

Imagine we have pictures of three animals: dogs, cats, and horses. We train our image classifier and get a confusion matrix, noticing that the model tends to predict that dogs are horses. But then we ...
user avatar
  • 32.4k
1 vote
1 answer
2k views

On which set (train/val/test) do people calculate F1 score, precision and recall?

This may be a stupid question, but when I was looking at the definition of precision/recall etc. it was not mentioned anywhere which set (training/validation/test) this metric should be calculated ...
user avatar
2 votes
0 answers
21 views

Is it harder to spoof (adversarial examples) a model trained to optimize a proper scoring rule than an improper scoring rule?

The way I figure, if we train a model to stick points on the correct side of a threshold like $0.5$, then all we have to do is tweak a $1$ with a predicted probability of $0.51$ to give a predicted ...
user avatar
  • 32.4k
1 vote
0 answers
47 views

Validation loss decreasing but accuracy increasing - early stop on accuracy?

I'm designing an image classifier for places365. The validation loss after the first epoch is lower than all other epochs (9.65 cross entropy loss, 11.07 after 10th epoch), but equally the validation ...
user avatar
  • 934
0 votes
1 answer
66 views

How to aggregate log-likelihood score of many models?

I have a process through which I estimate the parameters of a model in order to make predictions. Through this process, I end up with many models (with different parameter values) that are then used ...
user avatar
  • 75
0 votes
0 answers
30 views

Sum of sensitivity and specificity meaningful?

I came across an article in the BMJ that claims that a useful rule of thumb metric for assessing a medical test's performance is that the sum of specificity and sensitivity should be greater than 1.5 ...
user avatar
  • 539
2 votes
1 answer
92 views

How to modify a logloss scoring rule for forecasted probabilities being a Beta distribution?

I've been reading up on scoring rules and posts such as this one Why is LogLoss preferred over other proper scoring rules? It's clear to me that if I have a set of biased coin and after examining each ...
user avatar
  • 353
0 votes
1 answer
251 views

validation accuracy, recall and precision remains constant after 30th epoch

I am using TensorFlow model EfficientNetB0 for transfer learning, but after a number of epochs the validation accuracy, -precision, and -recall remains constant. Is this something I should be worried ...
user avatar
1 vote
0 answers
38 views

Sensitivity, Accuracy, AUROC, Gini

I got following chart: The algorithms have been applied to a dataset where an outcome is pretty rare, it happens 10% of the times (binary, 0- 90%, 1-10%). It is the response whether a client is going ...
user avatar
2 votes
0 answers
52 views

In Bayesian modeling, are there any approaches to score the confidence of the predicted distribution, instead of just the point estimate?

In a situation where one is predicting a distribution, rather than just a point estimate, what are some approaches to check the goodness of fit on observed data? The goal here would be to get a metric ...
user avatar
  • 75
7 votes
1 answer
466 views

Circular reasoning in Harrell BBR 18-19?

I am looking at chapter 18 (Information Loss) of Harrell's "Biostatistics for Biomedical Reserach": https://hbiostat.org/doc/bbr.pdf. The example of 18-19 seems like circular reasoning. He ...
user avatar
  • 32.4k
5 votes
2 answers
92 views

logistic regression: compare performance to human prediction

Say I want to predict whether a patient develops a disorder or not. I have two prediction 'models': Clinicians estimating the probability of a patient developing the disorder and logistic regression ...
user avatar
  • 115
1 vote
1 answer
109 views

Find covariance of estimator and derivative of the log-likelihood function

Problem: Given and estimator $\hat k$. The estimation method is unknown (so, it can be max. likelihood, method of moments or another method), however, we know that $bias(\hat k) = 0$. Let $L$ be the ...
user avatar
  • 27
1 vote
0 answers
36 views

Forecast scores, proper scores and strictly proper scores

Let $\mu_F$ and $\sigma_F^2$ be the expectation and variance of a forecast model $F$ for observations $y$, and let $p_F(\cdot)$ denote the probability density function for the forecast model. Also let ...
user avatar
2 votes
1 answer
169 views

Can I measure the accuracy of a range of quantiles in my forecast distribution?

I am forecasting items and measuring the point forecast and distribution accuracy of numerous different models against actuals. To measure distribution accuracy I am using the continuous-ranked ...
user avatar
  • 31
1 vote
1 answer
102 views

About computation of Brier score

Assume that we have some count data $x_{1}, \dots, x_{n}$, generated by probability mass function $\textbf{p} = \{p_{1}, \dots, p_{s} \}$. Let $\hat{\theta}$ be some estimator of $\textbf{p}$. In ...
user avatar
  • 426
1 vote
1 answer
190 views

Intuition behind Brier score weighing step for censored data

Sources seem to suggest that when calculating Brier scores involving right-censored data, one must weigh the otherwise mean square error function with the inverse probability of censoring weights ...
user avatar
1 vote
1 answer
425 views

F1 weighted vs. Log loss in SciKit learn RandomSearchCV

I am sorry to ask another question regarding this topic but I am still puzzled about the following: When I use 'F1_weighted' as my scoring argument in a ...
user avatar
  • 151
0 votes
1 answer
208 views

Scoring rules (log-loss vs. F1-weighted) and RandomizedSearchCV

I read multiple posts about scoring rules during cross-validation and the fact that the log-loss score is a proper scoring rule and, correct me if I am wrong, any threshold based approach is a ...
user avatar
  • 151
0 votes
1 answer
66 views

Are there constraints for the variance of predicted probability on calibrated models?

I'm sorry if the title is too vague. I'm not really sure of what I ask, this is a somewhat speculative question... The setting is that I'm using XGBoost in a binary classification problem (40% ...
user avatar

1
2 3 4 5