The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Questions tagged [scoring-rules]

Scoring rules are used to assess the accuracy of predicted probabilities, or more generally of predictive densities. Examples of scoring rules include the logarithmic, Brier, spherical, ranked probability and the Dawid-Sebastiani score and the predictive deviance.

Filter by
Sorted by
Tagged with
0
votes
0answers
11 views

Brier Score - Mean vs Median?

I have a probabilistic model for three distinct outcomes and I'm using the Brier Score to asses the predictive accuracy against another forecasting model. One of the models has a lower average Brier ...
1
vote
0answers
16 views

Quantile regression and mean log predictive score

During my research, I got stuck with an evaluation of my quantile regression models (qr). I have two QR models that model some economic variable in time. I would like to evaluate which model is better ...
2
votes
0answers
18 views

Prove that AUROC is an improper scoring rule [duplicate]

It has been stated in many places that AUROC is an improper scoring rule.But I haven't seen anyone proving it. Does someone have a working example that shows that maximizing AUROC actually moves away ...
3
votes
1answer
69 views

Are there any mathematical features that an evaluation metric must have?

I'm trying to optimize the hyperparameters of my model using the Bayesian approach with the hyperopt library. I have to code a ...
0
votes
0answers
19 views

Evaluating a model with Log Loss

I have been looking at alternative ways to intuitively understand the "goodness" of probability predictions from 2-class logistic regression models (and other ML classification models) and came ...
3
votes
2answers
224 views

Minimizing expected brier score and Brier score interpretation

For a probabilistic binary forecast, the BS (Brier score) is given by $$ \text{BS}= \begin{cases} (1-f_i)^2\\ f_i^2\\ \end{cases} $$ Where $f$ is the forecast. If the event occurs with probability $...
0
votes
0answers
15 views

Deciles Threshold Training vs Validation

I will try to be as clear as possible. I train a scoring model and the goal is to target top deciles for example from it (let's say, 10, 9 and 8). So i have my model, i apply on new datasets and now ...
3
votes
4answers
62 views

How to make sense out of integration over discrete data points?

Looking for a proof of the expected value of the score function equating zero, I came to this document that was recommended in another answer. Considering that we have a sample of n x_i values, I ...
0
votes
0answers
54 views

Evaluating predictive uncertainty classification models

I've been using BART (Bayesian Additive Regression Trees) for both regression and classification problems. BART, unlike many other tree based models, provides you with uncertainties on its predictions....
0
votes
0answers
26 views

Open Set Classification : How to count True Positive, False Positive,

TL;DR : How to compute TP,FP,TN,FN in Open Set Classification setting ? Even if the problem is simple; the answer may be tricky, so is my question. Given two sets of clusters : one from experiments, ...
1
vote
1answer
32 views

Scoring rules for count models on: training data vs. validation data

In order to evaluate and compare count models (e.g. Poisson regression), we can calculate scoring rules (e.g. Brier Score, Dawid-Sebastiani score, etc.) which are explained here: Error metrics for ...
0
votes
0answers
63 views

Interpretation of a brier score

Suppose that multiple brier scores were computed for two models $A,B$ and the density of the scores plotted as Where the average of $A$ is less than $B$ What would the interpretation of this be? ...
5
votes
1answer
76 views

How to deal with variables that are only relevant for some people? [duplicate]

I am reviewing an article. I can't be specific, but it involves validating a test for a health condition. Their goal is to come up with a score for risk of the condition. One variable is pregnancy. ...
4
votes
1answer
377 views

How to Compute the Brier Score for more than Two Classes

tl;dr How do I correctly compute the Brier score for more than two classes? I got confusing results with different approaches. Details below. As suggested to me in a comment to this question, I ...
1
vote
1answer
94 views

If $X\sim\text{Beta}(\theta,1)$, obtain the confidence interval of $100(1-\alpha)\%$ based on the asymptotic distribution of the score function

Let $X_{1},X_{2},\ldots,X_{n}$ be a random sample whose distribution is given by $\text{Beta}(\theta,1)$. Obtain the approximate confidence interval of $100(1-\alpha)\%$ based on the asymptotic ...
1
vote
1answer
69 views

Brier Score of a Prediction, Mathematical Notation

I am currently working on a logistic regression model that is fitted on the base of a training set ($D_0$) and is used to predict the outcome (0 or 1) of an independent test set ($D_1$). As an ...
1
vote
1answer
73 views

How to validate models beyond checking for overfitting

I have an unusual problem, which is that my model is performing too well and I am struggling to trust it. The data is a table of "snapshots" about moments in games of chess. For example, a game that ...
1
vote
0answers
44 views

Brier scores and integrated brier scores

Whats the difference between brier scores and integrated brier scores conceptually and mathematically? When would you use one over the other?
0
votes
0answers
15 views

Clearness Scoring Function

Hello to the community, I would love to hear your opinion about a Prediction Model I am currently working on for my thesis, in order to predict the Clearness of a website. I created a dataset of 5....
1
vote
0answers
75 views

How do you / can you compare Bayesian vs. frequentist regression models?

I am working on a regression model to predict a target variable in a dataset with over 100 features. Three different regression models are defined and fit in order to compare their performance using $...
0
votes
0answers
19 views

How to develop a score with training and testing set (with R)

I would like to build a score with data that have the following characteristics: There are much more controls than case there are several variable with more or less differences between the 2 groups ...
1
vote
0answers
69 views

Probability score for Hierarchical classification models

We've a hierarchical classification system in place; where each level produces predictions with a probability. Here's how the hierarchy is setup Top level: 1 model; ~25 classes Level 1: 25 models(=25 ...
2
votes
1answer
320 views

Variance of reparameterization trick and score function

For a function $\mathbf E_{z\sim q_\phi(z|x)}[f(z)]$(assuming $f$ is continuous), where $q_\phi$ is a Gaussian distribution, if we want to compute the gradient w.r.t. $\phi$, we have two way to do ...
0
votes
0answers
27 views

Meaing for MOAC in Spherical Payoff

I want to implement this metric Spherical Payoff mentioned in both articles and Netica software to validate my bayesian network (through a test dataset), here are the formula that I got from my ...
3
votes
0answers
184 views

Using the Brier Score with ordinal data [duplicate]

I have a model built to predict an ordinal response with 3 levels (win, draw or lose, say) and I would like to evaluate predictive accuracy. Is it appropriate to use the Brier Score: $$ \sum_{i=1}^{...
0
votes
0answers
21 views

Determining the log-loss score for matrices that consist of three probabilities (Win/Draw/Loss)

I have a CSV-file with my own estimated probabilities for the outcome of a soccer match. There are three possible outcomes: a win, a draw and a loss. In order to determine how accurate my estimates ...
-2
votes
1answer
139 views

Calculate score for multiple variables [closed]

My question is similar to this. I have a large geographical area divided into small cells (approximately 2.25 km$^2$). I'm calculating some value (heat) for individual cells dynamically. It is an ...
2
votes
1answer
151 views

How to compare model fit of OLS and Poisson regression?

I have built two regression models to predict sales of different products based on a number of explanatory variables, with an offset term for the number of days each product was on sale. One is a ...
1
vote
0answers
79 views

What is null distribution of brier's score or logarithmic score? How to test them properly?

I would like to get some p-values for predictions in an independent test set according to proper scoring rules. So I would like to know if my brier score/logarithmic score is statistically ...
0
votes
0answers
84 views

Comparing goodness of fit of quasi-Poisson regression model predictions vs. unknown forecast

I have built a quasi-Poisson regression to predict sales of different products based on a number of explanatory variables, with an offset term for the number of days each product was on sale. To ...
12
votes
1answer
1k views

Is accuracy an improper scoring rule in a binary classification setting?

I have recently been learning about proper scoring rules for probabilistic classifiers. Several threads on this website have made a point of emphasizing that accuracy is an improper scoring rule and ...
1
vote
0answers
33 views

Interpreting score function in Cox model

Several sources state that the score function for the likelihood of a cox model is $$ \dfrac{\partial{}l(\beta)}{\partial\beta}=\Big(X_{i}\delta_i^T-\sum\limits_{i=1}^{n}\delta_i\dfrac{\sum\limits_{j\...
3
votes
1answer
104 views

Can you overfit with proper scoring rules, e.g., Brier score?

I have read a lot suggestions and literature about using Brier score to measure model performance. It seems to be likened to the holy grail of model evaluation metrics because it is a proper scoring ...
7
votes
2answers
58 views

Evaluating quality of predicted distributions

I have a set of data points $X_i, y_i$ where $x$ are the independent variables and I believe each $y_i$ can be modeled as being drawn from a exponential distributions with parameters $\lambda_i$. If ...
1
vote
0answers
258 views

XGBClassifier default scoring metric

I am working with pythons xgboost XGBClassifier on a multiclass classification problem. I am trying to interpret the score that sklearns ...
1
vote
0answers
530 views

Inconsistent results calculating the integrated brier score in R

I would like to calculate the integrated brier score as a measure of model performance for a cox survival model I am fitting. There are multiple packages and functions to do this: survcomp (sbrier....
16
votes
1answer
537 views

What does it mean that AUC is a semi-proper scoring rule?

A proper scoring rule is a rule that is maximized by a 'true' model and it doesn't allow 'hedging' or gaming the system (deliberately reporting different results as is the true belief of the model to ...
2
votes
0answers
147 views

Implementaiton of Continuous Ranked Probability Score (CRPS) when Observation is a Distribution

The most general form of the Continuous Ranked Probability Score (CRPS) is defined as, $\int_{\mathbb{R}} \big( \hat{F}^e(x) - F^0(x)\big)^2dx,$ for some true distribution, $F^0$, and empirical ...
0
votes
0answers
39 views

Scoring rule for comparing two distribution of probabilities

I need help with figuring out a proper scoring rule for the following task. There are 11 possible outcomes. There is a true probability distribution over these outcomes (known to me but not you). ...
4
votes
0answers
399 views

Are Log Predictive Likelihood, Log Predictive Probability, Log Marginal Likelihood and Log Predictive Density same?

I have seen different papers use different terms to express the scoring rules that they used to compare Bayesian models. Some of those terms are, Log Predictive Density (Bayesian Data Analysis - by ...
0
votes
1answer
425 views

How does the logarithmic scoring rule work given that it's undefined for zero?

[I'm not a mathematician, so please forgive any misuse of terminology] One way of understanding scoring rules is that they measure the 'distance' between the truth value of a statement, and the ...
0
votes
1answer
81 views

Algorithm to score aptitude test to yield highest possible correlations with outcome

My problem concerns an aptitude test containing a set of single choice items ($x_1 x_2 .. x_n$). For each item, the participant may have selected option 1 to 5. These choices are scored dichotomously (...
6
votes
1answer
87 views

In GLMs, why do we solve score(beta)=0 instead of just minimizing the negative log-likelihood?

When we search for a numerical way to find $\hat{\beta}$ in a GLM (say, a logistic regression), we could do a numerical optimization (minimization) of the negative log-likelihood. But instead, we go ...
107
votes
7answers
36k views

Why is accuracy not the best measure for assessing classification models?

This is a general question that was asked indirectly multiple times in here, but it lacks a single authoritative answer. It would be great to have a detailed answer to this for the reference. ...
1
vote
1answer
48 views

Non-Symmetry in Stability Index

A very popular index to measure the stability of characteristics of a scorecard is defined by the following formula: $$SI = \frac{1}{n} \sum \text{(actual in %)-(expected in %)} \cdot \log\left(\frac{...
0
votes
1answer
398 views

SVC doing great on validation & test data but scored very low on predicted data

First of all, this is my first machine learning project after taking Andrew Ng's course, so please bear with me. I'm working on the most famous dataset, the Titanic data. First, I split the dataset ...
1
vote
0answers
29 views

Verification on probabilities and interpretation of evaluation scores

my situation is the following: I have a matrix consists of purchase probabilities for different products per user. Retrospectively I have another matrix consists of real-purchases. Now my task is to ...
1
vote
0answers
27 views

How to combine purchase and click data togehter in sparse matrix

my problem is the following: I have purchase probability estimations of different products. The model behind don't take care of any inter-correlations through these products. So my task is to re-...
6
votes
3answers
3k views

Multi-lable classification - Brier Score or Log Loss?

I'm using scikit package with RandomForestClassifier, trying to predict binary or multi-lable classifications. I'm looking for a way to estimate the reliability of the model but really can't figure ...
2
votes
0answers
167 views

Can a calibrated prediction model that is able to discriminate have a poor Brier score?

I did a logistic regression with selected covariates on a dataset with about 10000 records and event rate of 10%. The cross validated c-index was 63% which admittedly is not very high. Looking at a ...