# Questions tagged [scoring-rules]

Scoring rules are used to assess the accuracy of predicted probabilities, or more generally of predictive densities. Examples of scoring rules include the logarithmic, Brier, spherical, ranked probability and the Dawid-Sebastiani score and the predictive deviance.

256 questions
Filter by
Sorted by
Tagged with
54 views

### Writing Kullback–Leibler divergence in terms of the score function

Assume we have two density functions $p(x)$ and $p'(x)$ for $x\in R^d$. I would find a connection between Kullback–Leibler divergence between two densities in terms of the difference between the ...
• 61
350 views

### What is the best epoch to evaluate the test images?

I created a training, a validation and a test set for an image classification task. Then, I did training using the training and did evaluation on validation set. So, the next step is to evaluate the ...
• 53
25 views

### Comparing performance of probabilistic regression models - how to adapt Brier score?

Suppose I have two predictions models, Model 1 and Model 2. I have a dataset containing observations, features and actual outcomes. For each observation, the “outcomes” (i.e. predictions) that the ...
• 101
37 views

### Evaluating estimator of expected value plus variation

I know that for a typical, the estimator can be evaluated based on the mean squared error (MSE) of the predictions. How can I evaluate an estimator that instead gives a value that is the prediction ...
160 views

• 19.4k
58 views

### Can we get probabilistic predictions evaluable by proper scoring rules from bayesian inference without evaluating the marginal likelihood?

Let's say we have a vector of inputs, $X=[x_0,\dots, x_{n-1}]$, and a vector of outputs, $Y=[y_0, \dots, y_{n-1}]$. We would like to predict the distribution of a new output ,$\hat{y}$, given a new ...
• 451
1 vote
120 views

### Calibrating CatBoostClassifier produces worse results

I'm performing multiclass probability prediction using CatBoostClassifier on a dataset with ~4000 rows, 13 features, 4 target classes. Dataset has outliers, but it is balanced. For this task I'm using ...
85 views

### Two-sided KS-Test for Evaluating Prediction Model?

In the article https://ginimachine.com/blog/machine-learning-model-evaluation/ there is a proposal of using Two-Sided KS-Tests for evaluating the accuracy of predictions from Machine Learning (ML) ...
• 133
136 views

### Why do we maximize likelihood (sum of logs) and not simply maximize sum of probabilities? [duplicate]

In logistic regression we find the maximum likelihood estimator - $\max \prod_{i} p(y_i \mid x_i)$. Which in practice means maximizing the sum of log likelihoods. This makes sense, I understand MLE. ...
310 views

### Reconciling optimisation for log-likelihood and Brier score

Both log-likelihood and Brier score are proper scoring rules. As such, they reach the optimum when the predicted probabilities match the true ones. Since there is only one true probability for each ...
• 9,468
121 views

### What is a scoring rule for binary classification that is not dependent on the "difficulty" of classification?

Consider a model that predicts the probability of some binary event $Y$ (potentially given some features $X$). Denote the estimated probability of $Y$ occurring as $\hat{p}$. One possible choice for a ...
• 441
72 views

### Error metric for regression of count data: Poisson Deviance or Mean Square Error?

I would like to understand what difference it makes, if I use, for example, either Mean Square Error or Poisson Deviance as error metric/loss function for a regression of count data. Are there any a-...
• 2,738
358 views

### How to rank data based on multiple variables

I need help in ranking data, says car models in this case, based on multiple variables. For some variables (eg. mpg), the higher the better. For some variables (eg. car age), the lower the better. For ...
• 79
53 views

### Can the Brier score and Concordance index be anti-correlated?

I am using a proportional hazards Cox model to predict the survival probability of some mechanical components. I am using a combined L1-L2 penalization, and I want to optimize the (integrated) Brier ...
93 views

### Multiclass proper scoring rule decomposition: (weighted) average across the categories?

I have found a Python function that calculates the decomposition of various proper scoring rule, such as Brier score and log loss. However, it does not seem to accept arrays as arguments, so if I want ...
• 65.8k
23 views

### How to show that the influence function of minimum density power divergence estimator with positive tuning parameter is bounded?

In the linked paper, in the influence function section, the term ${u_{\theta}(y)}{f_{\theta}(y)}^\alpha$ is directly called bounded which i do not get the explanation of? Here $\alpha > 0$ is the ...
426 views

### As Brier Score = MSE, does MSE in a regression have a calibration-discrimination decomposition?

When the outcome of a supervised learning problem is binary and probabilities are predicted, Brier score can be decomposed into a measure of calibration and a measure of discrimination. ...
• 65.8k
124 views

### Is Brier score strictly proper in multi-label problems?

In problems where one of $3+$ categories can be observed and we prodict the probability of each category being observed, it is known that the Brier score is a strictly proper scoring rule that is ...
• 65.8k
98 views

### Why isn't there a square root version of the Brier score similar to how RMSE complements MSE?

When computing the mean squared error of a regression model, we get a metric in square units. For ease of interpretation, we can therefore instead compute the root mean squared error, which are in ...
1 vote
30 views

### Is it preferred to evaluate with a metric at a single decision threshold (eg Fbeta) vs averageing across thresholds (eg ROC-AUC)

Consider these two approaches to evaluating a classifiers performance: Choose a metric that summarizes the confusion matrix at a pre-determined decision threshold. Common suggestions seems to be ...
163 views

### What is the relationship between the Brier score "refinement" and the area under the ROC curve?

In the Wikipedia article on Brier score, there is a claim that the "refinement" in the two-component decomposition of Brier score is related to the area under the receiver-operator ...
• 65.8k