The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Questions tagged [scoring-rules]

Scoring rules are used to assess the accuracy of predicted probabilities, or more generally of predictive densities. Examples of scoring rules include the logarithmic, Brier, spherical, ranked probability and the Dawid-Sebastiani score and the predictive deviance.

127 questions
Filter by
Sorted by
Tagged with
11 views

Brier Score - Mean vs Median?

I have a probabilistic model for three distinct outcomes and I'm using the Brier Score to asses the predictive accuracy against another forecasting model. One of the models has a lower average Brier ...
16 views

Quantile regression and mean log predictive score

During my research, I got stuck with an evaluation of my quantile regression models (qr). I have two QR models that model some economic variable in time. I would like to evaluate which model is better ...
18 views

Prove that AUROC is an improper scoring rule [duplicate]

It has been stated in many places that AUROC is an improper scoring rule.But I haven't seen anyone proving it. Does someone have a working example that shows that maximizing AUROC actually moves away ...
69 views

Are there any mathematical features that an evaluation metric must have?

I'm trying to optimize the hyperparameters of my model using the Bayesian approach with the hyperopt library. I have to code a ...
19 views

Evaluating a model with Log Loss

I have been looking at alternative ways to intuitively understand the "goodness" of probability predictions from 2-class logistic regression models (and other ML classification models) and came ...
224 views

19 views

How to develop a score with training and testing set (with R)

I would like to build a score with data that have the following characteristics: There are much more controls than case there are several variable with more or less differences between the 2 groups ...
69 views

Probability score for Hierarchical classification models

We've a hierarchical classification system in place; where each level produces predictions with a probability. Here's how the hierarchy is setup Top level: 1 model; ~25 classes Level 1: 25 models(=25 ...
320 views

Variance of reparameterization trick and score function

For a function $\mathbf E_{z\sim q_\phi(z|x)}[f(z)]$(assuming $f$ is continuous), where $q_\phi$ is a Gaussian distribution, if we want to compute the gradient w.r.t. $\phi$, we have two way to do ...
27 views

Meaing for MOAC in Spherical Payoff

I want to implement this metric Spherical Payoff mentioned in both articles and Netica software to validate my bayesian network (through a test dataset), here are the formula that I got from my ...
184 views

104 views

Can you overfit with proper scoring rules, e.g., Brier score?

I have read a lot suggestions and literature about using Brier score to measure model performance. It seems to be likened to the holy grail of model evaluation metrics because it is a proper scoring ...
58 views

Evaluating quality of predicted distributions

I have a set of data points $X_i, y_i$ where $x$ are the independent variables and I believe each $y_i$ can be modeled as being drawn from a exponential distributions with parameters $\lambda_i$. If ...
258 views

XGBClassifier default scoring metric

I am working with pythons xgboost XGBClassifier on a multiclass classification problem. I am trying to interpret the score that sklearns ...
530 views

Inconsistent results calculating the integrated brier score in R

I would like to calculate the integrated brier score as a measure of model performance for a cox survival model I am fitting. There are multiple packages and functions to do this: survcomp (sbrier....
537 views

What does it mean that AUC is a semi-proper scoring rule?

A proper scoring rule is a rule that is maximized by a 'true' model and it doesn't allow 'hedging' or gaming the system (deliberately reporting different results as is the true belief of the model to ...
147 views

Implementaiton of Continuous Ranked Probability Score (CRPS) when Observation is a Distribution

The most general form of the Continuous Ranked Probability Score (CRPS) is defined as, $\int_{\mathbb{R}} \big( \hat{F}^e(x) - F^0(x)\big)^2dx,$ for some true distribution, $F^0$, and empirical ...
39 views

Scoring rule for comparing two distribution of probabilities

I need help with figuring out a proper scoring rule for the following task. There are 11 possible outcomes. There is a true probability distribution over these outcomes (known to me but not you). ...
399 views

Are Log Predictive Likelihood, Log Predictive Probability, Log Marginal Likelihood and Log Predictive Density same?

I have seen different papers use different terms to express the scoring rules that they used to compare Bayesian models. Some of those terms are, Log Predictive Density (Bayesian Data Analysis - by ...
425 views

How does the logarithmic scoring rule work given that it's undefined for zero?

[I'm not a mathematician, so please forgive any misuse of terminology] One way of understanding scoring rules is that they measure the 'distance' between the truth value of a statement, and the ...
81 views

Algorithm to score aptitude test to yield highest possible correlations with outcome

My problem concerns an aptitude test containing a set of single choice items ($x_1 x_2 .. x_n$). For each item, the participant may have selected option 1 to 5. These choices are scored dichotomously (...
87 views

In GLMs, why do we solve score(beta)=0 instead of just minimizing the negative log-likelihood?

When we search for a numerical way to find $\hat{\beta}$ in a GLM (say, a logistic regression), we could do a numerical optimization (minimization) of the negative log-likelihood. But instead, we go ...
36k views

Why is accuracy not the best measure for assessing classification models?

This is a general question that was asked indirectly multiple times in here, but it lacks a single authoritative answer. It would be great to have a detailed answer to this for the reference. ...
48 views

Non-Symmetry in Stability Index

A very popular index to measure the stability of characteristics of a scorecard is defined by the following formula: SI = \frac{1}{n} \sum \text{(actual in %)-(expected in %)} \cdot \log\left(\frac{...
398 views

SVC doing great on validation & test data but scored very low on predicted data

First of all, this is my first machine learning project after taking Andrew Ng's course, so please bear with me. I'm working on the most famous dataset, the Titanic data. First, I split the dataset ...
29 views

Verification on probabilities and interpretation of evaluation scores

my situation is the following: I have a matrix consists of purchase probabilities for different products per user. Retrospectively I have another matrix consists of real-purchases. Now my task is to ...
27 views

How to combine purchase and click data togehter in sparse matrix

my problem is the following: I have purchase probability estimations of different products. The model behind don't take care of any inter-correlations through these products. So my task is to re-...