Linked Questions
10 questions linked to/from When is it appropriate to use an improper scoring rule?
2
votes
1
answer
528
views
Custom metrics for multiclass classification when class errors have different weights
I have a multiclass classification problem (eg. the target variable is made by 4 different outcomes: Product A, Product B, Product C and NO Product). Not all the errors are equal: for example, if the ...
CommunityBot
- 1
5
votes
1
answer
83
views
Penalising Error above a certain Threshold
I have a ML model (a NN in the specific but I don't think it's important for the purpose of my question) that is doing pretty decent at his job, which is predicting the demand of a certain substance X ...
- 111k
226
votes
10
answers
121k
views
Why is accuracy not the best measure for assessing classification models?
This is a general question that was asked indirectly multiple times in here, but it lacks a single authoritative answer. It would be great to have a detailed answer to this for the reference.
...
- 980
8
votes
2
answers
526
views
Is there a Good Illustrative Example where the Hinge Loss (SVM) Gives a Higher Accuracy than the Logistic Loss
Vladimir Vapnik wrote:
“When solving a problem of interest, do not solve a more general
problem as an intermediate step. Try to get the answer that you really
need but not a more general one.”
...
- 31.3k
2
votes
0
answers
44
views
Testing for Statistical Significance of 200 million Features [closed]
I have 200 million features and 1 label (features and label have about 1 million observations). Features are binary, and each has an unknown but different amounts of True and False. Label is also ...
- 153
28
votes
1
answer
2k
views
What does it mean that AUC is a semi-proper scoring rule?
A proper scoring rule is a rule that is maximized by a 'true' model and it doesn't allow 'hedging' or gaming the system (deliberately reporting different results as is the true belief of the model to ...
- 41k
30
votes
2
answers
4k
views
What is the statistical model behind the SVM algorithm?
I have learned that, when dealing with data using model-based approach, the first step is modeling data procedure as a statistical model. Then the next step is developing efficient/fast inference/...
- 557
5
votes
2
answers
4k
views
Classification accuracy based on probability
Let's say we have a simple binary classification problem. So for a predictor X we want to predict response Y. Y is binary, so either 0 or 1. Now let's say we use two different classifiers, model1 and ...
- 34.9k
2
votes
0
answers
162
views
Assessing Classification Accuracy with False Positives and False Negatives
I have been reading this forum but cannot find anything specific enough to address my problem.
I have classified disease in the below image (red spots), and verified disease by GPS (Red Circles).
...
- 100k
1
vote
0
answers
72
views
When to use predictive power versus when to use model fitting metrics?
I built a binary classifier using logistic regression. But I can't seem to rationalize this in my mind. After cross validation, the model's AUC is 0.9003. But, as a sanity check, I ran a GOF (...
- 2,209