3
$\begingroup$

When evaluating a model, for example a binary classifier, should the train and test set have 50% + and 50% - label distribution or could the distribution be random?

If the distribution is biased in the train/test sets e.g., 80% + and 20%-, the precision/recall scores may not be representative. For example, the model may do well on classifying positive points but may misclassify a lot of negative points. It's recall is high but its precision could still be high because there aren't too many false positives because there are less negative points in the dataset.

Is AUC robust metric against such imbalanced distributions? Or is it best to balance the distribution in train/test data in order to compute more accurate precision and recall values?

I read this Kaggle forum post: Precision-recall AUC vs ROC AUC for class imbalance problems, but it doesn't discuss the issue I'm raising about dataset distribution.

$\endgroup$
  • $\begingroup$ What is the larger context here? Do you have a particular fitted model & want to know if it's good, or are you asking about abstract properties of precision / recall vs sensitivity / specificity, eg? Are you worried about what happens when you train a model on an imbalanced dataset & use it latter w/ a balanced dataset? BTW, how can you have a dataset w/ 70% + & 20% -? $\endgroup$ – gung - Reinstate Monica Jul 26 '16 at 19:16
  • 1
    $\begingroup$ oops, fixed that. I am worried about training on an imbalanced dataset, evaluating on an imbalanced dataset and then using it on a possibly balanced dataset. $\endgroup$ – bla345 Jul 26 '16 at 19:21
  • $\begingroup$ If you choose the evaluation metric wisely, I see no problem. $\endgroup$ – Firebug Jul 26 '16 at 20:35
1
$\begingroup$

It is important to understand that improper accuracy scoring rules lead to bogus models and are affected by imbalance (the AUROC (c-index) being one improper scoring rule that is an exception; the c-index is independent of outcome prevalence but is not sensitive enough to be used as an optimality criterion). Use of full probability estimators and proper scoring rules are recipes for success, and handle extreme imbalance. This issues are expanded upon here and here.

$\endgroup$
0
$\begingroup$

If you modify the prevalence of your training set (thereby the prevalence of your CV partitions for model evaluation and selection) you essentially change how the model fits your data (e.g. internally minimizes the error of the fit). This could both cause be beneficial for your goal, or be degradation - that depends on the details of your problem and goal. If you do so, the one important thing is to not modify the prevalence of your held-back test set.

To give an example: if there are unbalanced classes and the model fits data in a way that the positive, less prominent class is badly predicted (bad TPR but good TNR), and you can't change this using hyperparameters, this could likely still be changed e.g. by down- or upsampling, weighting of samples, etc. The internal change is that thereby the less represented samples each contribute a bigger part to the error during fitting, hence how the model fits and represents data.

Besides changing how models fit the data, as you pointed out, one could change the error measure, which largely depends on what you want to achieve. This can be beneficial if you are evaluating and selecting one model from multiple candidate models. For example, using TPR and TNR (+ ROC AUC etc.) boils down to optimizing for correctly predicting samples as positive P samples, if they are actually P - as well as negative N samples, if they are actually N. This essentially leaves out class prevalence (e.g. many false positives don't degrade true positives or false negatives). In contrast, precision and recall (+ precision-recall ROC) look at the same ground information differently, as false positives directly influence both recall and precision.

So, bottom line: it depends on what you want to achieve. There are problems/approaches out there that use artificially manipulated training prevalence to change how the model represented training data - e.g. better representing certain classes - which in turn better satisfies predefined goals. But, as pointed out in the beginning, it's important to not change the prevalence of your final test set, because this one reflects the TPR, TNR, precision, recall, and whichever other metric you look at for the final model on data with (hopefully) natural prevalence.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.