10 questions linked to/from When is it appropriate to use an improper scoring rule?
Why is accuracy not the best measure for assessing classification models?
This is a general question that was asked indirectly multiple times in here, but it lacks a single authoritative answer. It would be great to have a detailed answer to this for the reference. ...
What is the statistical model behind the SVM algorithm?
I have learned that, when dealing with data using model-based approach, the first step is modeling data procedure as a statistical model. Then the next step is developing efficient/fast inference/...
Is there a Good Illustrative Example where the Hinge Loss (SVM) Gives a Higher Accuracy than the Logistic Loss
Vladimir Vapnik wrote: “When solving a problem of interest, do not solve a more general problem as an intermediate step. Try to get the answer that you really need but not a more general one.” ...
Classification accuracy based on probability
Let's say we have a simple binary classification problem. So for a predictor X we want to predict response Y. Y is binary, so either 0 or 1. Now let's say we use two different classifiers, model1 and ...
What does it mean that AUC is a semi-proper scoring rule?
A proper scoring rule is a rule that is maximized by a 'true' model and it doesn't allow 'hedging' or gaming the system (deliberately reporting different results as is the true belief of the model to ...
Penalising Error above a certain Threshold
I have a ML model (a NN in the specific but I don't think it's important for the purpose of my question) that is doing pretty decent at his job, which is predicting the demand of a certain substance X ...
Custom metrics for multiclass classification when class errors have different weights
I have a multiclass classification problem (eg. the target variable is made by 4 different outcomes: Product A, Product B, Product C and NO Product). Not all the errors are equal: for example, if the ...
Testing for Statistical Significance of 200 million Features [closed]
I have 200 million features and 1 label (features and label have about 1 million observations). Features are binary, and each has an unknown but different amounts of True and False. Label is also ...
Assessing Classification Accuracy with False Positives and False Negatives
I have been reading this forum but cannot find anything specific enough to address my problem. I have classified disease in the below image (red spots), and verified disease by GPS (Red Circles). ...
When to use predictive power versus when to use model fitting metrics?
I built a binary classifier using logistic regression. But I can't seem to rationalize this in my mind. After cross validation, the model's AUC is 0.9003. But, as a sanity check, I ran a GOF (...