Linked Questions
12 questions linked to/from When is it appropriate to use an improper scoring rule?
245
votes
11
answers
137k
views
Why is accuracy not the best measure for assessing classification models?
This is a general question that was asked indirectly multiple times in here, but it lacks a single authoritative answer. It would be great to have a detailed answer to this for the reference.
...
- 140k
7
votes
3
answers
2k
views
How much of neural network overconfidence in predictions can be attributed to modelers optimizing threshold-based metrics?
Neural network "classifiers" output probability scores, and when they are optimized via crossentropy loss (common) or another proper scoring rule, they are optimized in expectation by the ...
- 65.8k
30
votes
2
answers
4k
views
What is the statistical model behind the SVM algorithm?
I have learned that, when dealing with data using model-based approach, the first step is modeling data procedure as a statistical model. Then the next step is developing efficient/fast inference/...
- 557
11
votes
2
answers
863
views
Is there a Good Illustrative Example where the Hinge Loss (SVM) Gives a Higher Accuracy than the Logistic Loss
Vladimir Vapnik wrote:
“When solving a problem of interest, do not solve a more general
problem as an intermediate step. Try to get the answer that you really
need but not a more general one.”
...
- 55.2k
5
votes
2
answers
4k
views
Classification accuracy based on probability
Let's say we have a simple binary classification problem. So for a predictor X we want to predict response Y. Y is binary, so either 0 or 1. Now let's say we use two different classifiers, model1 and ...
29
votes
1
answer
3k
views
What does it mean that AUC is a semi-proper scoring rule?
A proper scoring rule is a rule that is maximized by a 'true' model and it doesn't allow 'hedging' or gaming the system (deliberately reporting different results as is the true belief of the model to ...
- 7,719
5
votes
1
answer
247
views
Penalising Error above a certain Threshold
I have a ML model (a NN in the specific but I don't think it's important for the purpose of my question) that is doing pretty decent at his job, which is predicting the demand of a certain substance X ...
- 327
2
votes
1
answer
760
views
Custom metrics for multiclass classification when class errors have different weights
I have a multiclass classification problem (eg. the target variable is made by 4 different outcomes: Product A, Product B, Product C and NO Product). Not all the errors are equal: for example, if the ...
- 213
2
votes
0
answers
137
views
When *is* classification accuracy the right measure of performance
Plenty has been discussed on Cross Validated about the drawbacks of classification accuracy when it comes to evaluating classification models. One good answer is here, for instance.
Under what ...
- 65.8k
2
votes
0
answers
45
views
Testing for Statistical Significance of 200 million Features [closed]
I have 200 million features and 1 label (features and label have about 1 million observations). Features are binary, and each has an unknown but different amounts of True and False. Label is also ...
- 153
2
votes
0
answers
163
views
Assessing Classification Accuracy with False Positives and False Negatives
I have been reading this forum but cannot find anything specific enough to address my problem.
I have classified disease in the below image (red spots), and verified disease by GPS (Red Circles).
...
- 75
1
vote
0
answers
76
views
When to use predictive power versus when to use model fitting metrics?
I built a binary classifier using logistic regression. But I can't seem to rationalize this in my mind. After cross validation, the model's AUC is 0.9003. But, as a sanity check, I ran a GOF (...
- 2,299