Timeline for Imbalanced Test Data
Current License: CC BY-SA 4.0
11 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Dec 6, 2018 at 13:49 | vote | accept | slaw | ||
| Dec 6, 2018 at 8:22 | comment | added | usεr11852 | @slaw: Please see my answer below. | |
| Dec 6, 2018 at 8:21 | answer | added | usεr11852 | timeline score: 2 | |
| Dec 6, 2018 at 6:00 | comment | added | Jacob H | @slaw tweaking the class weights is a good idea. Also, you might want to change up your learner, AUC or accuracy around .7 is not very good. If my math is right, your only classifying 85% of you're majority class correctly, which is also not great. Good Luck! | |
| Dec 6, 2018 at 1:31 | comment | added | slaw | In my case, the model is only getting about 6% of the minority class correct for the test set and it's probably what I care about more. I might have to adjust the class weights when training the model's loss function. | |
| Dec 6, 2018 at 1:26 | comment | added | Jacob H | @slaw generally speaking, a data set is considered to be imbalanced when the ratio of minority to majority is closer to 1:100, not 1:5. I've seen learners do a good job on data with an imbalance of 1:1000, without explicitly correcting for imbalance. | |
| Dec 6, 2018 at 1:13 | comment | added | slaw | @usεr11852: Can you explain what you mean by "calibrate our predicted probabilities in the down/upsampled space"? Links to other discussions, examples, or references would really help me understand this and why it is important. Thanks! | |
| Dec 6, 2018 at 1:11 | comment | added | slaw |
@JacobH: Can you elaborate on what you mean by imbalance is not going to be an issue? It performs about the same without oversampling
|
|
| Dec 6, 2018 at 0:27 | comment | added | usεr11852 | There a number of threads on learning from imbalanced data on CV you may want to look at them for suggestions. That being said: 1. 1-to-5 is not really imbalanced, 2. if we choose to significantly alter the proportions of classes in our data we must calibrate our predicted probabilities in the downsampled/upsampled space. | |
| Dec 6, 2018 at 0:24 | comment | added | Jacob H | Generally, in my experience, as along as 1%-2% of the sample is comprised of the minority class, imbalance is not going to be an issue. How does the model do without oversampling? Also what does the in and out sample confusion matrix look like? | |
| Dec 5, 2018 at 23:42 | history | asked | slaw | CC BY-SA 4.0 |