0
$\begingroup$

I am currently working on an imbalanced data-set (1% of 1). However I am a bit concerned by the underlying model.

I treated the problem as a classification problem, making some hypotheses on the distinguishability of the classes, I have been using different set of classifier, with good AUC (up to 0.9).

But the theory suggest that each instance as a low probability of being 1 with each instance characteristic changing the probability (say from 0.1% to 10%). In other terms, I have a rare event model. In this approach an instance with output 0 would be nearly indistinguishable from an instance with output 1. I feel like regression techniques should be used.

Does a rare event model, instead of a rare class one, invalidate the classification approach ? Or the 0.9 AUC is good enough so that the classification approach should hold ?

|cite|improve this question
$\endgroup$
  • $\begingroup$ I recommend using area under Precision & Recall curve as opposed to AUROC in the existence of highclass imbalance. This link explains why in detail. $\endgroup$ – Zhubarb Apr 25 at 15:12
1
$\begingroup$

Do not use accuracy to evaluate a classifier: Why is accuracy not the best measure for assessing classification models? Also Is accuracy an improper scoring rule in a binary classification setting? Everything in those threads applies equally to AUC. Instead, use proper scoring rules on probabilistic predictions. See also Are unbalanced datasets problematic, and (how) does oversampling (purport to) help?

|cite|improve this answer
$\endgroup$
  • $\begingroup$ (1) I was under the impression that the use of AUC (or F1-score) would allievate standard problems of using accuracy (as tending to label everything as the majority class). Do you have sources that explain AUC limitations simply ? (2) How do that translate in terms of classification v.s. regression approaches for rare event models ? (Or : Is the plan to calculate a proper score for the classification method, see it's bad and use regression instead ?) $\endgroup$ – were_cat Apr 25 at 10:04
  • $\begingroup$ (1) AUC is somewhat less bad than accuracy. Frank Harrell's posts are most enlightening, especially this one, which also contains a pointer to his book. $\endgroup$ – Stephan Kolassa Apr 25 at 10:24
  • $\begingroup$ (2) Are you using "regression" in the statistical or in the ML sense? I don't see a useful distinction between classification and regression models. What I would use would be a probabilistic model for the data you have, which seems to be binary, so something like logistic regression or an appropriate random forest that is capable of outputting predicted probabilities. Then assess your model using proper scoring rules. (This last assessment step works identically whatever your outcomes' data type is.) $\endgroup$ – Stephan Kolassa Apr 25 at 10:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.