Linked Questions

1 vote
0 answers

Why over/under-sampling could not help my model fitting? [duplicate]

I fit random forest to my imbalanced dataset with minority class 1. I found that the AUC under the imbalanced data was better than that of re-sampled dataset (over/under sampling). Can someone help to ...
LUSAQX's user avatar
  • 453
101 votes
8 answers

When is unbalanced data really a problem in Machine Learning?

We already had multiple questions about unbalanced data when using logistic regression, SVM, decision trees, bagging and a number of other similar questions, what makes it a very popular topic! ...
Tim's user avatar
  • 130k
64 votes
7 answers

Binary classification with strongly unbalanced classes

I have a data set in the form of (features, binary output 0 or 1), but 1 happens pretty rarely, so just by always predicting 0, I get accuracy between 70% and 90% (depending on the particular data I ...
LazyCat's user avatar
  • 842
105 votes
4 answers

Are unbalanced datasets problematic, and (how) does oversampling (purport to) help?

TL;DR See title. Motivation I am hoping for a canonical answer along the lines of "(1) No, (2) Not applicable, because (1)", which we can use to close many wrong questions about unbalanced ...
Stephan Kolassa's user avatar
7 votes
2 answers

Removing duplicates before train test split

Let's say you have a dataset generated from real world sampling which has lots of duplicates (the dependent and independent variables are identical) and you want to train a classifier to predict the ...
Ushnish's user avatar
  • 71
8 votes
1 answer

random forest for imbalanced data?

I have a dataset where yes=77 and no=16000, a highly imbalanced dataset. My plan was to identify the most important variables influencing the response variable using random forest and then develop a ...
MSilvy's user avatar
  • 139
8 votes
1 answer

When is oversampling poor practice?

For my particular domain and problem, I have data on the entire population. However, my "event" only occurs in 0.5% of the cases. I want my model to be able to pick up on significant characteristics ...
blacksite's user avatar
  • 644
3 votes
1 answer

Imbalanced Test Data

I have an imbalanced (1:5) training and test set with only two classes and have oversampled the training set with SMOTE so that the class ratio is 1:1. The ML model gives values over 0.7 for accuracy, ...
slaw's user avatar
  • 504
0 votes
1 answer

How do I perform a logistic regression w/ SMOTE

I want to understand which variables lead to an infection by parasites in a tree. Hence, I want to use stepwise logistic regression based on AIC. First, I describe what I would do, and then my code ...
Thomas's user avatar
  • 448
3 votes
0 answers

Effects of class imbalance on nn batch training

Say I have a binary classification task, where the positive class (1) is only 1% of the whole data set. Intuitively I can understand why this could be bad for the classifier as the model may learn ...
Sam's user avatar
  • 377
1 vote
0 answers

Why would random forest perform bad on unbalanced class

There is a huge number of posts saying that an imbalanced classes are bad. And only half explains it in terms of recall-presicion scores, meaning that accuracy can be high but F1 score low. What I ...
Alina's user avatar
  • 1,135
1 vote
1 answer

Prediction for imbalanced and small sample sized data

I have to create a classification model where my dataset contains 697 observations which only 18 are from the group of interest. As usual, I split data the into a training and test set stratified by ...
luizgg's user avatar
  • 111
-1 votes
1 answer

Imbalance Classification : How SMOTE handles majority class?

I'm working on Imbalance Classification problem with minority class(0.017%). I've read that imbalance classification can be handled using Undersampling, Oversampling and SMOTE. Major drawback of ...
siddharthabingi's user avatar
1 vote
1 answer

What are other ways of doing oversampling apart from SMOTE?

I have just begun learning about machine learning techniques and started solving problems on kaggle. I have a few questions about how to handle class imbalance: How to handle imbalance dataset ...
Heman473710's user avatar
1 vote
0 answers

Neural network for imbalanced data

I have an imbalanced data (n = 600, about 97% majority and 3% minority) with 20 features and a binary outcome. The data has been split into a training set and a test set (80%/20%). I used H2o autoML ...
user145331's user avatar

15 30 50 per page