Questions tagged [scoring-rules]

Scoring rules are used to assess the accuracy of predicted probabilities, or more generally of predictive densities. Examples of scoring rules include the logarithmic, Brier, spherical, ranked probability and the Dawid-Sebastiani score and the predictive deviance.

Filter by
Sorted by
Tagged with
0
votes
0answers
10 views

Evaluating a model with Log Loss

I have been looking at alternative ways to intuitively understand the "goodness" of probability predictions from 2-class logistic regression models (and other ML classification models) and came ...
3
votes
2answers
169 views

Minimizing expected brier score and Brier score interpretation

For a probabilistic binary forecast, the BS (Brier score) is given by $$ \text{BS}= \begin{cases} (1-f_i)^2\\ f_i^2\\ \end{cases} $$ Where $f$ is the forecast. If the event occurs with probability $...
0
votes
0answers
13 views

Deciles Threshold Training vs Validation

I will try to be as clear as possible. I train a scoring model and the goal is to target top deciles for example from it (let's say, 10, 9 and 8). So i have my model, i apply on new datasets and now ...
3
votes
4answers
47 views

How to make sense out of integration over discrete data points?

Looking for a proof of the expected value of the score function equating zero, I came to this document that was recommended in another answer. Considering that we have a sample of n x_i values, I ...
0
votes
0answers
29 views

Evaluating predictive uncertainty classification models

I've been using BART (Bayesian Additive Regression Trees) for both regression and classification problems. BART, unlike many other tree based models, provides you with uncertainties on its predictions....
0
votes
0answers
17 views

Open Set Classification : How to count True Positive, False Positive,

TL;DR : How to compute TP,FP,TN,FN in Open Set Classification setting ? Even if the problem is simple; the answer may be tricky, so is my question. Given two sets of clusters : one from experiments, ...
1
vote
1answer
26 views

Scoring rules for count models on: training data vs. validation data

In order to evaluate and compare count models (e.g. Poisson regression), we can calculate scoring rules (e.g. Brier Score, Dawid-Sebastiani score, etc.) which are explained here: Error metrics for ...
0
votes
0answers
52 views

Interpretation of a brier score

Suppose that multiple brier scores were computed for two models $A,B$ and the density of the scores plotted as Where the average of $A$ is less than $B$ What would the interpretation of this be? ...
5
votes
1answer
73 views

How to deal with variables that are only relevant for some people? [duplicate]

I am reviewing an article. I can't be specific, but it involves validating a test for a health condition. Their goal is to come up with a score for risk of the condition. One variable is pregnancy. ...
4
votes
1answer
246 views

How to Compute the Brier Score for more than Two Classes

tl;dr How do I correctly compute the Brier score for more than two classes? I got confusing results with different approaches. Details below. As suggested to me in a comment to this question, I ...
1
vote
1answer
79 views

If $X\sim\text{Beta}(\theta,1)$, obtain the confidence interval of $100(1-\alpha)\%$ based on the asymptotic distribution of the score function

Let $X_{1},X_{2},\ldots,X_{n}$ be a random sample whose distribution is given by $\text{Beta}(\theta,1)$. Obtain the approximate confidence interval of $100(1-\alpha)\%$ based on the asymptotic ...
1
vote
1answer
62 views

Brier Score of a Prediction, Mathematical Notation

I am currently working on a logistic regression model that is fitted on the base of a training set ($D_0$) and is used to predict the outcome (0 or 1) of an independent test set ($D_1$). As an ...
1
vote
1answer
72 views

How to validate models beyond checking for overfitting

I have an unusual problem, which is that my model is performing too well and I am struggling to trust it. The data is a table of "snapshots" about moments in games of chess. For example, a game that ...
1
vote
0answers
40 views

Brier scores and integrated brier scores

Whats the difference between brier scores and integrated brier scores conceptually and mathematically? When would you use one over the other?
0
votes
0answers
15 views

Clearness Scoring Function

Hello to the community, I would love to hear your opinion about a Prediction Model I am currently working on for my thesis, in order to predict the Clearness of a website. I created a dataset of 5....
1
vote
0answers
69 views

How do you / can you compare Bayesian vs. frequentist regression models?

I am working on a regression model to predict a target variable in a dataset with over 100 features. Three different regression models are defined and fit in order to compare their performance using $...
0
votes
0answers
19 views

How to develop a score with training and testing set (with R)

I would like to build a score with data that have the following characteristics: There are much more controls than case there are several variable with more or less differences between the 2 groups ...
1
vote
0answers
64 views

Probability score for Hierarchical classification models

We've a hierarchical classification system in place; where each level produces predictions with a probability. Here's how the hierarchy is setup Top level: 1 model; ~25 classes Level 1: 25 models(=25 ...
2
votes
1answer
266 views

Variance of reparameterization trick and score function

For a function $\mathbf E_{z\sim q_\phi(z|x)}[f(z)]$(assuming $f$ is continuous), where $q_\phi$ is a Gaussian distribution, if we want to compute the gradient w.r.t. $\phi$, we have two way to do ...
0
votes
0answers
23 views

Meaing for MOAC in Spherical Payoff

I want to implement this metric Spherical Payoff mentioned in both articles and Netica software to validate my bayesian network (through a test dataset), here are the formula that I got from my ...
3
votes
0answers
160 views

Using the Brier Score with ordinal data [duplicate]

I have a model built to predict an ordinal response with 3 levels (win, draw or lose, say) and I would like to evaluate predictive accuracy. Is it appropriate to use the Brier Score: $$ \sum_{i=1}^{...
0
votes
0answers
21 views

Determining the log-loss score for matrices that consist of three probabilities (Win/Draw/Loss)

I have a CSV-file with my own estimated probabilities for the outcome of a soccer match. There are three possible outcomes: a win, a draw and a loss. In order to determine how accurate my estimates ...
-2
votes
1answer
79 views

Calculate score for multiple variables [closed]

My question is similar to this. I have a large geographical area divided into small cells (approximately 2.25 km$^2$). I'm calculating some value (heat) for individual cells dynamically. It is an ...
2
votes
1answer
124 views

How to compare model fit of OLS and Poisson regression?

I have built two regression models to predict sales of different products based on a number of explanatory variables, with an offset term for the number of days each product was on sale. One is a ...
1
vote
0answers
78 views

What is null distribution of brier's score or logarithmic score? How to test them properly?

I would like to get some p-values for predictions in an independent test set according to proper scoring rules. So I would like to know if my brier score/logarithmic score is statistically ...
0
votes
0answers
74 views

Comparing goodness of fit of quasi-Poisson regression model predictions vs. unknown forecast

I have built a quasi-Poisson regression to predict sales of different products based on a number of explanatory variables, with an offset term for the number of days each product was on sale. To ...
10
votes
1answer
957 views

Is accuracy an improper scoring rule in a binary classification setting?

I have recently been learning about proper scoring rules for probabilistic classifiers. Several threads on this website have made a point of emphasizing that accuracy is an improper scoring rule and ...
1
vote
0answers
33 views

Interpreting score function in Cox model

Several sources state that the score function for the likelihood of a cox model is $$ \dfrac{\partial{}l(\beta)}{\partial\beta}=\Big(X_{i}\delta_i^T-\sum\limits_{i=1}^{n}\delta_i\dfrac{\sum\limits_{j\...
3
votes
1answer
94 views

Can you overfit with proper scoring rules, e.g., Brier score?

I have read a lot suggestions and literature about using Brier score to measure model performance. It seems to be likened to the holy grail of model evaluation metrics because it is a proper scoring ...
7
votes
2answers
58 views

Evaluating quality of predicted distributions

I have a set of data points $X_i, y_i$ where $x$ are the independent variables and I believe each $y_i$ can be modeled as being drawn from a exponential distributions with parameters $\lambda_i$. If ...
1
vote
0answers
244 views

XGBClassifier default scoring metric

I am working with pythons xgboost XGBClassifier on a multiclass classification problem. I am trying to interpret the score that sklearns ...
1
vote
0answers
459 views

Inconsistent results calculating the integrated brier score in R

I would like to calculate the integrated brier score as a measure of model performance for a cox survival model I am fitting. There are multiple packages and functions to do this: survcomp (sbrier....
14
votes
1answer
455 views

What does it mean that AUC is a semi-proper scoring rule?

A proper scoring rule is a rule that is maximized by a 'true' model and it doesn't allow 'hedging' or gaming the system (deliberately reporting different results as is the true belief of the model to ...
2
votes
0answers
127 views

Implementaiton of Continuous Ranked Probability Score (CRPS) when Observation is a Distribution

The most general form of the Continuous Ranked Probability Score (CRPS) is defined as, $\int_{\mathbb{R}} \big( \hat{F}^e(x) - F^0(x)\big)^2dx,$ for some true distribution, $F^0$, and empirical ...
0
votes
0answers
37 views

Scoring rule for comparing two distribution of probabilities

I need help with figuring out a proper scoring rule for the following task. There are 11 possible outcomes. There is a true probability distribution over these outcomes (known to me but not you). ...
3
votes
0answers
364 views

Are Log Predictive Likelihood, Log Predictive Probability, Log Marginal Likelihood and Log Predictive Density same?

I have seen different papers use different terms to express the scoring rules that they used to compare Bayesian models. Some of those terms are, Log Predictive Density (Bayesian Data Analysis - by ...
0
votes
1answer
379 views

How does the logarithmic scoring rule work given that it's undefined for zero?

[I'm not a mathematician, so please forgive any misuse of terminology] One way of understanding scoring rules is that they measure the 'distance' between the truth value of a statement, and the ...
0
votes
1answer
77 views

Algorithm to score aptitude test to yield highest possible correlations with outcome

My problem concerns an aptitude test containing a set of single choice items ($x_1 x_2 .. x_n$). For each item, the participant may have selected option 1 to 5. These choices are scored dichotomously (...
6
votes
1answer
84 views

In GLMs, why do we solve score(beta)=0 instead of just minimizing the negative log-likelihood?

When we search for a numerical way to find $\hat{\beta}$ in a GLM (say, a logistic regression), we could do a numerical optimization (minimization) of the negative log-likelihood. But instead, we go ...
100
votes
7answers
33k views

Why is accuracy not the best measure for assessing classification models?

This is a general question that was asked indirectly multiple times in here, but it lacks a single authoritative answer. It would be great to have a detailed answer to this for the reference. ...
1
vote
1answer
48 views

Non-Symmetry in Stability Index

A very popular index to measure the stability of characteristics of a scorecard is defined by the following formula: $$SI = \frac{1}{n} \sum \text{(actual in %)-(expected in %)} \cdot \log\left(\frac{...
0
votes
1answer
356 views

SVC doing great on validation & test data but scored very low on predicted data

First of all, this is my first machine learning project after taking Andrew Ng's course, so please bear with me. I'm working on the most famous dataset, the Titanic data. First, I split the dataset ...
1
vote
0answers
28 views

Verification on probabilities and interpretation of evaluation scores

my situation is the following: I have a matrix consists of purchase probabilities for different products per user. Retrospectively I have another matrix consists of real-purchases. Now my task is to ...
1
vote
0answers
27 views

How to combine purchase and click data togehter in sparse matrix

my problem is the following: I have purchase probability estimations of different products. The model behind don't take care of any inter-correlations through these products. So my task is to re-...
6
votes
3answers
3k views

Multi-lable classification - Brier Score or Log Loss?

I'm using scikit package with RandomForestClassifier, trying to predict binary or multi-lable classifications. I'm looking for a way to estimate the reliability of the model but really can't figure ...
2
votes
0answers
162 views

Can a calibrated prediction model that is able to discriminate have a poor Brier score?

I did a logistic regression with selected covariates on a dataset with about 10000 records and event rate of 10%. The cross validated c-index was 63% which admittedly is not very high. Looking at a ...
1
vote
1answer
91 views

How to see how good my probability estimations are?

I am estimating the probability for an event to happen based on certain criteria prior to the event. The event either happens or does not happen (1 or 0). I have a few hundred thousand such pairs of 1)...
2
votes
0answers
182 views

Why is LogLoss preferred over other proper scoring rules?

It seems anytime people care about estimating probabilities accurately they choose LogLoss as the evaluation metric. But there are many other evaluation metrics which will prefer accurate estimation ...
0
votes
2answers
355 views

Comparing overlap and matches in different datasets

My goal: a measure/test/score/etc. that can give me a numerical value for how similar two datasets are. Let's say I have two datasets, each with a differing number of datapoints, where each datapoint ...
2
votes
0answers
488 views

Confidence interval for Brier score in R

I am using the lrm function in the rms package in R to fit a logistic regression model. The function outputs various useful stats, including Brier score. Is it possible to output the confidence ...