I have a binary classificaiton problem with positive label proportion is very small, 1% at most of the cases. So, recall, precision and f score for the positive label is used to evauluate my classifiers.
However, the commom overfitting detecting method uses the metric MSE, accuracy to detect the situtaion. MSE, accuracy is useless in my situation.
So, is there any methods to detecting overfitting in the small proporation positive label binary classification problem?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ Do you have enough positive label results to do cross validation? $\endgroup$– russellpierceCommented Dec 25, 2015 at 16:03
-
$\begingroup$ I have 10,000,000+ samples, so 1% mean 100,000+ positive labels, I think it enougth $\endgroup$– bourneliCommented Dec 26, 2015 at 8:46
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
The “gold standard” is log loss (“crossentropy loss” in some circles).
$$ \text{Log Loss}\\ L(y,\hat p)= -\dfrac{1}{N}\sum\left( y_i\log(\hat p_i) + (1-y_i)\log(1-\hat p_i) \right) $$
The $\hat p=(\hat p_1,\dots ,\hat p_N)$ is the vector of predicted probabilities, not the classifications based on a threshold.
Like the log loss, Brier score is a strictly proper scoring rule that assesses the predicted probabilities instead of categorical classifications, and it is equivalent to MSE. Thus, I disagree that MSE is useless in this setting.
Despite the issues with accuracy, precision, recall, and F-score, if you have reason to believe those to be the metrics of interest, those could be viable. If you dislike such metrics in this situation because basically every prediction is the same category (with $99$:$1$ imbalance, every predicted probability might be below the software-default cutoff of $0.5$, and such predictions would not be surprising), then these metrics are revealing issues in your predictions that you want to catch and correct before you use the model for real.
