Linked Questions

1
vote
1answer
32 views

Problem of unbalanced data

unbalanced data is an issue that can effect the performnce of classification model ,several remides can be done to balance the data two of them are upsampling and downsampling , my questions is : how ...
25
votes
4answers
3k views

What problem does oversampling, undersampling, and SMOTE solve?

In a recent, well recieved, question, Tim asks when is unbalanced data really a problem in Machine Learning? The premise of the question is that there is a lot of machine learning literature ...
0
votes
1answer
24 views

Binary Classification problem for imbalanced dataset

I am new to machine learning and need help. I have a dataset with two classes(0,1) where is 0 is Profitable and 1 is Unprofitable. Ratio of 0:1 in train set is 150/52 Taking positive as "1"(...
0
votes
0answers
23 views

Deal with imbalanced data [duplicate]

Building machine learning models to do forecast, sometimes the dataset was used is imbalanced, and there are some methods to deal with this issue such as the resample method and choose other metrics(...
1
vote
0answers
28 views

Running SMOTE on very large class imbalanced datasets - batched or subsampled implementations

There is a theoretical and computational aspect to this question. I was trying to use SMOTE to reduce class imbalance in a rather large dataset--about 8 million rows. The data has a binary outcome ...
4
votes
4answers
2k views

How to handle data imbalance in classification? [duplicate]

I am working on a text classification problem. My data is highly imbalanced. For example, one category has 700 documents while the other has 30. I have around 30 categories. I tried different ...
0
votes
1answer
20 views

logic behind balancing? [duplicate]

I am a newbie in stats, and while reading: https://towardsdatascience.com/having-an-imbalanced-dataset-here-is-how-you-can-solve-it-1640568947eb I don't seem to understand why is an imbalanced ...
6
votes
2answers
246 views

Should we balance the data set if the data is intrinsically unbalanced? [duplicate]

Say I want to predict the cancer rate(regression)/predict the whether a person has cancer or not(classification). The data intrinsically has few cancer patients/low cancer rate, say 1/200. And the ...
0
votes
0answers
23 views

After applying SMOTE, the class distribution doesn't match the real world. Is this a problem? [duplicate]

I have an extremely unbalanced dataset with two classes: 1: 1,800 # class 1 0: 40,000 # class 0 This is real world customer data of churned/not churned If I ...
1
vote
0answers
17 views

When is oversampling preferable to undersampling and vice versa?

When data is unbalanced, that is, when the distribution of classes being predicted is very uneven (e.g. 90%/10% for two classes or 10%/15%/75% for three classes), many machine learning models have ...
34
votes
3answers
2k views

What is the root cause of the class imbalance problem?

I've been thinking a lot about the "class imbalance problem" in machine/statistical learning lately, and am drawing ever deeper into a feeling that I just don't understand what is going on. First let ...
119
votes
7answers
42k views

Why is accuracy not the best measure for assessing classification models?

This is a general question that was asked indirectly multiple times in here, but it lacks a single authoritative answer. It would be great to have a detailed answer to this for the reference. ...
1
vote
3answers
176 views

Binary classification with imbalanced classes [duplicate]

I have a manufacture dataset of 65 million rows corresponding to 65 millions distinct items. Out of those 65 millions, I have 60,000 of them that failed a certain test, thus I have very imbalanced ...
0
votes
0answers
12 views

Random sampling for both under- and over-representative class

I have an unbalanced dataset. Let's say I have 500 positives and 50,000 negatives. Can I deal with this by randomly choose 300 out of 500 positives and also 300 out of 50,000 negatives? Does this ...
1
vote
3answers
278 views

Does oversampling/undersampling change the distribution of the data?

I have an imbalanced dataset (10000 positives and 300 negatives) and have divided this into train and test sets. I perform oversampling/undersampling only on the train set since doing this on the test ...

15 30 50 per page