Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [scoring-rules]

Scoring rules are used to assess the accuracy of predicted probabilities, or more generally of predictive densities. Examples of scoring rules include the logarithmic, Brier, spherical, ranked probability and the Dawid-Sebastiani score and the predictive deviance.

1
vote
1answer
47 views

How to validate models beyond checking for overfitting

I have an unusual problem, which is that my model is performing too well and I am struggling to trust it. The data is a table of "snapshots" about moments in games of chess. For example, a game that ...
0
votes
0answers
24 views

Brier scores and integrated brier scores

Whats the difference between brier scores and integrated brier scores conceptually and mathematically? When would you use one over the other?
0
votes
0answers
14 views

Clearness Scoring Function

Hello to the community, I would love to hear your opinion about a Prediction Model I am currently working on for my thesis, in order to predict the Clearness of a website. I created a dataset of 5....
1
vote
0answers
48 views

How do you / can you compare Bayesian vs. frequentist regression models?

I am working on a regression model to predict a target variable in a dataset with over 100 features. Three different regression models are defined and fit in order to compare their performance using $...
0
votes
0answers
14 views

How to develop a score with training and testing set (with R)

I would like to build a score with data that have the following characteristics: There are much more controls than case there are several variable with more or less differences between the 2 groups ...
0
votes
0answers
22 views

Probability score for Hierarchical classification models

We've a hierarchical classification system in place; where each level produces predictions with a probability. Here's how the hierarchy is setup Top level: 1 model; ~25 classes Level 1: 25 models(=25 ...
1
vote
1answer
71 views

Variance of reparameterization trick and score function

For a function $\mathbf E_{z\sim q_\phi(z|x)}[f(z)]$(assuming $f$ is continuous), where $q_\phi$ is a Gaussian distribution, if we want to compute the gradient w.r.t. $\phi$, we have two way to do ...
0
votes
0answers
14 views

Meaing for MOAC in Spherical Payoff

I want to implement this metric Spherical Payoff mentioned in both articles and Netica software to validate my bayesian network (through a test dataset), here are the formula that I got from my ...
3
votes
0answers
71 views

Using the Brier Score with ordinal data [duplicate]

I have a model built to predict an ordinal response with 3 levels (win, draw or lose, say) and I would like to evaluate predictive accuracy. Is it appropriate to use the Brier Score: $$ \sum_{i=1}^{...
0
votes
0answers
16 views

Determining the log-loss score for matrices that consist of three probabilities (Win/Draw/Loss)

I have a CSV-file with my own estimated probabilities for the outcome of a soccer match. There are three possible outcomes: a win, a draw and a loss. In order to determine how accurate my estimates ...
-2
votes
1answer
37 views

Calculate score for multiple variables [closed]

My question is similar to this. I have a large geographical area divided into small cells (approximately 2.25 km$^2$). I'm calculating some value (heat) for individual cells dynamically. It is an ...
2
votes
0answers
70 views

How to compare model fit of OLS and Poisson regression?

I have built two regression models to predict sales of different products based on a number of explanatory variables, with an offset term for the number of days each product was on sale. One is a ...
0
votes
0answers
44 views

What is null distribution of brier's score or logarithmic score? How to test them properly?

I would like to get some p-values for predictions in an independent test set according to proper scoring rules. So I would like to know if my brier score/logarithmic score is statistically ...
0
votes
0answers
80 views

pseudo R2 as xgboost objective function

I want to use a custom objective function with xgboost: 1 - (log(y) - log(p)) / (log(y) - log(q)) y = true value, p = my probabilities, q = some other base ...
0
votes
0answers
34 views

Comparing goodness of fit of quasi-Poisson regression model predictions vs. unknown forecast

I have built a quasi-Poisson regression to predict sales of different products based on a number of explanatory variables, with an offset term for the number of days each product was on sale. To ...
7
votes
1answer
474 views

Is accuracy an improper scoring rule in a binary classification setting?

I have recently been learning about proper scoring rules for probabilistic classifiers. Several threads on this website have made a point of emphasizing that accuracy is an improper scoring rule and ...
1
vote
0answers
30 views

Interpreting score function in Cox model

Several sources state that the score function for the likelihood of a cox model is $$ \dfrac{\partial{}l(\beta)}{\partial\beta}=\Big(X_{i}\delta_i^T-\sum\limits_{i=1}^{n}\delta_i\dfrac{\sum\limits_{j\...
1
vote
1answer
64 views

Can you overfit with proper scoring rules, e.g., Brier score?

I have read a lot suggestions and literature about using Brier score to measure model performance. It seems to be likened to the holy grail of model evaluation metrics because it is a proper scoring ...
7
votes
2answers
54 views

Evaluating quality of predicted distributions

I have a set of data points $X_i, y_i$ where $x$ are the independent variables and I believe each $y_i$ can be modeled as being drawn from a exponential distributions with parameters $\lambda_i$. If ...
1
vote
0answers
202 views

XGBClassifier default scoring metric

I am working with pythons xgboost XGBClassifier on a multiclass classification problem. I am trying to interpret the score that sklearns ...
0
votes
0answers
23 views

Log predictive score when bootstrapping

How do I evaluate the log predictive score $$ LPS(y_0)=\log f(y_0|x) $$ when I have $B$ bootstrap replications as a proxy for $f$?
1
vote
0answers
348 views

Inconsistent results calculating the integrated brier score in R

I would like to calculate the integrated brier score as a measure of model performance for a cox survival model I am fitting. There are multiple packages and functions to do this: survcomp (sbrier....
14
votes
1answer
327 views

What does it mean that AUC is a semi-proper scoring rule?

A proper scoring rule is a rule that is maximized by a 'true' model and it doesn't allow 'hedging' or gaming the system (deliberately reporting different results as is the true belief of the model to ...
2
votes
0answers
106 views

Implementaiton of Continuous Ranked Probability Score (CRPS) when Observation is a Distribution

The most general form of the Continuous Ranked Probability Score (CRPS) is defined as, $\int_{\mathbb{R}} \big( \hat{F}^e(x) - F^0(x)\big)^2dx,$ for some true distribution, $F^0$, and empirical ...
1
vote
0answers
104 views

comparing two estimators [closed]

I tried to solve the problem of comparing two estimators for soccer matches. The "estimators" are actually two punters trying two predict games results. The predicted value is between 0 and 1. The ...
0
votes
0answers
29 views

Scoring rule for comparing two distribution of probabilities

I need help with figuring out a proper scoring rule for the following task. There are 11 possible outcomes. There is a true probability distribution over these outcomes (known to me but not you). ...
3
votes
0answers
265 views

Are Log Predictive Likelihood, Log Predictive Probability, Log Marginal Likelihood and Log Predictive Density same?

I have seen different papers use different terms to express the scoring rules that they used to compare Bayesian models. Some of those terms are, Log Predictive Density (Bayesian Data Analysis - by ...
0
votes
1answer
298 views

How does the logarithmic scoring rule work given that it's undefined for zero?

[I'm not a mathematician, so please forgive any misuse of terminology] One way of understanding scoring rules is that they measure the 'distance' between the truth value of a statement, and the ...
0
votes
1answer
56 views

Algorithm to score aptitude test to yield highest possible correlations with outcome

My problem concerns an aptitude test containing a set of single choice items ($x_1 x_2 .. x_n$). For each item, the participant may have selected option 1 to 5. These choices are scored dichotomously (...
6
votes
1answer
74 views

In GLMs, why do we solve score(beta)=0 instead of just minimizing the negative log-likelihood?

When we search for a numerical way to find $\hat{\beta}$ in a GLM (say, a logistic regression), we could do a numerical optimization (minimization) of the negative log-likelihood. But instead, we go ...
0
votes
0answers
21 views

Scorecard logic to measure performance of managers

I need to create a scorecard where I can compare and contrast 3 different managers on one single measures/metric. The problem I'm having is to come up with a defend-able and fair scorecard logic/...
86
votes
7answers
23k views

Why is accuracy not the best measure for assessing classification models?

This is a general question that was asked indirectly multiple times in here, but it lacks a single authoritative answer. It would be great to have a detailed answer to this for the reference. ...
1
vote
1answer
46 views

Non-Symmetry in Stability Index

A very popular index to measure the stability of characteristics of a scorecard is defined by the following formula: $$SI = \frac{1}{n} \sum \text{(actual in %)-(expected in %)} \cdot \log\left(\frac{...
0
votes
1answer
244 views

SVC doing great on validation & test data but scored very low on predicted data

First of all, this is my first machine learning project after taking Andrew Ng's course, so please bear with me. I'm working on the most famous dataset, the Titanic data. First, I split the dataset ...
1
vote
0answers
25 views

Verification on probabilities and interpretation of evaluation scores

my situation is the following: I have a matrix consists of purchase probabilities for different products per user. Retrospectively I have another matrix consists of real-purchases. Now my task is to ...
1
vote
0answers
27 views

How to combine purchase and click data togehter in sparse matrix

my problem is the following: I have purchase probability estimations of different products. The model behind don't take care of any inter-correlations through these products. So my task is to re-...
6
votes
3answers
2k views

Multi-lable classification - Brier Score or Log Loss?

I'm using scikit package with RandomForestClassifier, trying to predict binary or multi-lable classifications. I'm looking for a way to estimate the reliability of the model but really can't figure ...
2
votes
0answers
131 views

Can a calibrated prediction model that is able to discriminate have a poor Brier score?

I did a logistic regression with selected covariates on a dataset with about 10000 records and event rate of 10%. The cross validated c-index was 63% which admittedly is not very high. Looking at a ...
1
vote
1answer
80 views

How to see how good my probability estimations are?

I am estimating the probability for an event to happen based on certain criteria prior to the event. The event either happens or does not happen (1 or 0). I have a few hundred thousand such pairs of 1)...
3
votes
0answers
171 views

Why is LogLoss preferred over other proper scoring rules?

It seems anytime people care about estimating probabilities accurately they choose LogLoss as the evaluation metric. But there are many other evaluation metrics which will prefer accurate estimation ...
0
votes
2answers
290 views

Comparing overlap and matches in different datasets

My goal: a measure/test/score/etc. that can give me a numerical value for how similar two datasets are. Let's say I have two datasets, each with a differing number of datapoints, where each datapoint ...
3
votes
0answers
380 views

Confidence interval for Brier score in R

I am using the lrm function in the rms package in R to fit a logistic regression model. The function outputs various useful stats, including Brier score. Is it possible to output the confidence ...
0
votes
1answer
733 views

What is wrong with my neural network?

I am building a machine learning model to attempt to predict the winner of a sports match based on historical statistics of the two teams. My model (a neural network) appears to get about 70% ...
1
vote
0answers
384 views

Correct use of a development (a.k.a. validation) set with PredefinedSplit and GridSearchCV

I'm dealing with a classical machine learning classification task. I'm using the sklearn library. My question is about the correct use of the test, development (...
4
votes
1answer
328 views

How to decide the “best” accuracy score for prediction of binary outcome?

Dr Frank Harrell mentioned in his book and BIOS 330 course that Accuracy score used to drive model building should be a continuous score that utilizes all the information in the data (e.g. Brier ...
3
votes
0answers
126 views

Assessing Classification Accuracy with False Positives and False Negatives

I have been reading this forum but cannot find anything specific enough to address my problem. I have classified disease in the below image (red spots), and verified disease by GPS (Red Circles). ...
10
votes
3answers
744 views

How do I choose the best metric to measure my calibration?

I program and do test-driven development. After I made a change in my code I run my tests. Sometimes they succeed and sometimes they fail. Before I run a test I write down a number from 0.01 to 0.99 ...
4
votes
2answers
643 views

Optimize a classification algorithm using mean per-class accuracy

I have a binary classification problem and am trying to find a way to optimize my machine learning algorithm using a performance metric based on the per-class error rate. If I'm not misinterpreting ...
1
vote
0answers
59 views

What is the benefit of developing different scores for LGD modelling?

In the LGD Model flow presented in the figure 4.13 in the book "Developing Credit Risk Models Using SAS Enterprise Miner and SAS/STAT: Theory and Application" which is partially available on the web: ...
25
votes
1answer
1k views

What is the root cause of the class imbalance problem?

I've been thinking a lot about the "class imbalance problem" in machine/statistical learning lately, and am drawing ever deeper into a feeling that I just don't understand what is going on. First let ...