Linked Questions

2 votes
0 answers

When *is* classification accuracy the right measure of performance

Plenty has been discussed on Cross Validated about the drawbacks of classification accuracy when it comes to evaluating classification models. One good answer is here, for instance. Under what ...
Dave's user avatar
  • 65k
5 votes
1 answer

Penalising Error above a certain Threshold

I have a ML model (a NN in the specific but I don't think it's important for the purpose of my question) that is doing pretty decent at his job, which is predicting the demand of a certain substance X ...
paolopazzo's user avatar
12 votes
2 answers

Is there a Good Illustrative Example where the Hinge Loss (SVM) Gives a Higher Accuracy than the Logistic Loss

Vladimir Vapnik wrote: “When solving a problem of interest, do not solve a more general problem as an intermediate step. Try to get the answer that you really need but not a more general one.” ...
Dikran Marsupial's user avatar
7 votes
3 answers

How much of neural network overconfidence in predictions can be attributed to modelers optimizing threshold-based metrics?

Neural network "classifiers" output probability scores, and when they are optimized via crossentropy loss (common) or another proper scoring rule, they are optimized in expectation by the ...
Dave's user avatar
  • 65k
2 votes
1 answer

Custom metrics for multiclass classification when class errors have different weights

I have a multiclass classification problem (eg. the target variable is made by 4 different outcomes: Product A, Product B, Product C and NO Product). Not all the errors are equal: for example, if the ...
A1010's user avatar
  • 213
2 votes
0 answers

Testing for Statistical Significance of 200 million Features [closed]

I have 200 million features and 1 label (features and label have about 1 million observations). Features are binary, and each has an unknown but different amounts of True and False. Label is also ...
Franc Weser's user avatar
29 votes
1 answer

What does it mean that AUC is a semi-proper scoring rule?

A proper scoring rule is a rule that is maximized by a 'true' model and it doesn't allow 'hedging' or gaming the system (deliberately reporting different results as is the true belief of the model to ...
rep_ho's user avatar
  • 7,709
30 votes
2 answers

What is the statistical model behind the SVM algorithm?

I have learned that, when dealing with data using model-based approach, the first step is modeling data procedure as a statistical model. Then the next step is developing efficient/fast inference/...
i_love_somebody's user avatar
243 votes
11 answers

Why is accuracy not the best measure for assessing classification models?

This is a general question that was asked indirectly multiple times in here, but it lacks a single authoritative answer. It would be great to have a detailed answer to this for the reference. ...
Tim's user avatar
  • 140k
5 votes
2 answers

Classification accuracy based on probability

Let's say we have a simple binary classification problem. So for a predictor X we want to predict response Y. Y is binary, so either 0 or 1. Now let's say we use two different classifiers, model1 and ...
Leander Moesinger's user avatar
2 votes
0 answers

Assessing Classification Accuracy with False Positives and False Negatives

I have been reading this forum but cannot find anything specific enough to address my problem. I have classified disease in the below image (red spots), and verified disease by GPS (Red Circles). ...
Alexander's user avatar
1 vote
0 answers

When to use predictive power versus when to use model fitting metrics?

I built a binary classifier using logistic regression. But I can't seem to rationalize this in my mind. After cross validation, the model's AUC is 0.9003. But, as a sanity check, I ran a GOF (...
makansij's user avatar
  • 2,289