10 questions linked to/from When is it appropriate to use an improper scoring rule?
Custom metrics for multiclass classification when class errors have different weights
I have a multiclass classification problem (eg. the target variable is made by 4 different outcomes: Product A, Product B, Product C and NO Product). Not all the errors are equal: for example, if the ...
Penalising Error above a certain Threshold
I have a ML model (a NN in the specific but I don't think it's important for the purpose of my question) that is doing pretty decent at his job, which is predicting the demand of a certain substance X ...
Why is accuracy not the best measure for assessing classification models?
This is a general question that was asked indirectly multiple times in here, but it lacks a single authoritative answer. It would be great to have a detailed answer to this for the reference. ...
Is there a Good Illustrative Example where the Hinge Loss (SVM) Gives a Higher Accuracy than the Logistic Loss
Vladimir Vapnik wrote: “When solving a problem of interest, do not solve a more general problem as an intermediate step. Try to get the answer that you really need but not a more general one.” ...
Testing for Statistical Significance of 200 million Features [closed]
I have 200 million features and 1 label (features and label have about 1 million observations). Features are binary, and each has an unknown but different amounts of True and False. Label is also ...
What does it mean that AUC is a semi-proper scoring rule?
A proper scoring rule is a rule that is maximized by a 'true' model and it doesn't allow 'hedging' or gaming the system (deliberately reporting different results as is the true belief of the model to ...
What is the statistical model behind the SVM algorithm?
I have learned that, when dealing with data using model-based approach, the first step is modeling data procedure as a statistical model. Then the next step is developing efficient/fast inference/...
Classification accuracy based on probability
Let's say we have a simple binary classification problem. So for a predictor X we want to predict response Y. Y is binary, so either 0 or 1. Now let's say we use two different classifiers, model1 and ...
Assessing Classification Accuracy with False Positives and False Negatives
I have been reading this forum but cannot find anything specific enough to address my problem. I have classified disease in the below image (red spots), and verified disease by GPS (Red Circles). ...
When to use predictive power versus when to use model fitting metrics?
I built a binary classifier using logistic regression. But I can't seem to rationalize this in my mind. After cross validation, the model's AUC is 0.9003. But, as a sanity check, I ran a GOF (...