Scoring rules are used to assess the accuracy of predicted probabilities, or more generally of predictive densities. Examples of scoring rules include the logarithmic, Brier, spherical, ranked probability and the Dawid-Sebastiani score and the predictive deviance.
0
votes
0answers
4 views
Quantitative score for a binary qualatative assesment
Following from this question: 900/1000 cars would it be possible to derive a quantitative score for the participants? For instance people getting it correct ('correct' being agreeing with the ...
0
votes
0answers
9 views
The best formula for combining two parameters with different priority
I have a pretty general question: I'm working on my thesis and I took an online test from some people. I want to assign score to each person based on
number of their correct answer (C) and
their ...
2
votes
2answers
45 views
Multi-lable classification - Brier Score or Log Loss?
I'm using scikit package with RandomForestClassifier, trying to predict binary or multi-lable classifications.
I'm looking for a way to estimate the reliability of the model but really can't figure ...
0
votes
0answers
14 views
How to calibrate purchase probabilities on additional data?
my problem is the following: I have purchase probabilitiy estimations of some products. The model behind don't take care of any inter-correlations through these products. So my task is to re-calibrate ...
0
votes
0answers
8 views
Problem for Running an Algorithm in a Loop and Replication
I am trying to run the following small simulation with R=1000 replications. Within each replication, there is a Scoring Algorithm with two unknown parameters (beta and sigma2nu). It takes too much ...
1
vote
0answers
22 views
Can a calibrated prediction model that is able to discriminate have a poor Brier score?
I did a logistic regression with selected covariates on a dataset with about 10000 records and event rate of 10%. The cross validated c-index was 63% which admittedly is not very high. Looking at a ...
1
vote
1answer
41 views
How to see how good my probability estimations are?
I am estimating the probability for an event to happen based on certain criteria prior to the event. The event either happens or does not happen (1 or 0). I have a few hundred thousand such pairs of 1)...
3
votes
0answers
73 views
Why is LogLoss preferred over other proper scoring rules?
It seems anytime people care about estimating probabilities accurately they choose LogLoss as the evaluation metric. But there are many other evaluation metrics which will prefer accurate estimation ...
0
votes
2answers
56 views
Comparing overlap and matches in different datasets
My goal: a measure/test/score/etc. that can give me a numerical value for how similar two datasets are.
Let's say I have two datasets, each with a differing number of datapoints, where each datapoint ...
3
votes
0answers
96 views
Confidence interval for Brier score in R
I am using the lrm function in the rms package in R to fit a logistic regression model. The function outputs various useful stats, including Brier score.
Is it possible to output the confidence ...
0
votes
0answers
67 views
Selection of model selection method
Suppose I am a professional bettor using a mathematical model to place bets on e.g. horse racing.
Suppose further that I have been using my model for quite a while and have saved records of my ...
0
votes
0answers
30 views
Presenting performance of multi-class classification
I have a biometric scheme that generates n x n (multi-class) confusion matrices, (1,1) is user1 authentication attempts matched to user1, (1,2) is user attempts matched to user2, and so forth. I'd ...
3
votes
0answers
55 views
Assessing Classification Accuracy with False Positives and False Negatives
I have been reading this forum but cannot find anything specific enough to address my problem.
I have classified disease in the below image (red spots), and verified disease by GPS (Red Circles).
...
10
votes
3answers
440 views
How do I choose the best metric to measure my calibration?
I program and do test-driven development. After I made a change in my code I run my tests. Sometimes they succeed and sometimes they fail. Before I run a test I write down a number from 0.01 to 0.99 ...
0
votes
0answers
38 views
What is the benefit of developing different scores for LGD modelling?
In the LGD Model flow presented in the figure 4.13 in the book "Developing Credit Risk Models Using SAS Enterprise Miner and SAS/STAT: Theory and Application" which is partially available on the web:
...
