Timeline for What is the root cause of the class imbalance problem?
Current License: CC BY-SA 3.0
8 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 17, 2018 at 6:02 | comment | added | Stephan Kolassa | This presumes implicitly (1) that the KPI we attempt to maximize is accuracy, and (2) that accuracy is an appropriate KPI for classification model evaluation. It isn't. | |
| Mar 19, 2017 at 14:43 | comment | added | usεr11852 | Try "cost sensitive training" it should have plenty of references. (Sorry reading some economics paper recently and my terminology gets a bit muddled, it is essentially the same. For references specifically using the term utility function for classification see here and here). We know that model $F$ won't be the same. Otherwise we wouldn't use techniques like ROSE, SMOTE or down-/up-sampling. | |
| Mar 19, 2017 at 14:29 | comment | added | Hugh Perkins | I googled for 'utility function', but nothing came up. Do you have a link/reference? I think from the context, what you are calling a 'utility function' is essentially the model $F$ above? Model $F$ is invariant across the various scenarios. One interesting question perhaps is, if one trains model $G$ directly, using unbalanced data, will the underlying, possibly implicit, model $F$ be similar/identical to a model $F$ trained, via training model $G$, on balanced data? | |
| Mar 19, 2017 at 12:03 | comment | added | usεr11852 |
I generally agree; I am not fully convinced about the proper scoring rule necessity but on the other hand the "actual purpose" of any classification model is the useful prediction of class membership, ie. you need an informed utility function. I would argue that generally for imbalanced problems assigning cost/gain to FP, TP, etc. is probably the best way to have a reasonable utility function; in the absence of relevant domain knowledge this can be hairy. I almost always use as my first choice Cohen's $k$, a somewhat conservative metric of "agreement", because of that reason.
|
|
| Mar 19, 2017 at 10:36 | comment | added | Hugh Perkins | I dont think one can say that the loss function is 'right' or 'wrong' without knowing the actual purpose of the model. If the goal is for the machine learning model to 'look cool/useful', then the $G^*$ model is better, but if it's to maximize eg scoring on some test/exam, where 99 of the questions have answer A, and one has answer B, and we only have a 90% chance of predicting the answer correctly, we're better off just choosing A for everything, and that's what the loss function above does. | |
| Mar 19, 2017 at 0:29 | comment | added | usεr11852 | Can you please try to make your answer a bit more particular to the questions being asked? While clearly thoughtful it reads mostly as commentary rather than an answer. For example, just for commentary purposes one could argue that using an improper scoring rule like the loss function defined is fundamentally wrong and therefore the subsequent analysis is invalid. | |
| Mar 18, 2017 at 23:25 | history | edited | Hugh Perkins | CC BY-SA 3.0 |
added 136 characters in body
|
| Mar 18, 2017 at 23:19 | history | answered | Hugh Perkins | CC BY-SA 3.0 |