Questions tagged [oversampling]

Sampling cases with differential probability, so that classes that occur rarely in the population occur more often in the training data. Does *not* address the problems in unbalanced classes.

Filter by
Sorted by
Tagged with
35
votes
0answers
865 views

Are unbalanced datasets problematic, and (how) does oversampling (purport to) help? [duplicate]

TL;DR See title. Motivation I am hoping for a canonical answer along the lines of "(1) No, (2) Not applicable, because (1)", which we can use to close many wrong questions about unbalanced datasets ...
22
votes
2answers
17k views

Testing Classification on Oversampled Imbalance Data

I am working on severely imbalanced data. In literature, several methods are used to re-balance the data using re-sampling (over- or under-sampling). Two good approaches are: SMOTE: Synthetic ...
20
votes
1answer
17k views

Opinions about Oversampling in general, and the SMOTE algorithm in particular [closed]

What is your opinion about oversampling in classification in general, and the SMOTE algorithm in particular? Why would we not just apply a cost/penalty to adjust for imbalance in class data and any ...
12
votes
1answer
16k views

ROSE and SMOTE oversampling methods

Can somebody give me a brief explanation of the differences between those two resampling methods : ROSE and SMOTE ?
11
votes
2answers
7k views

Sampling with replacement in R randomForest

The randomForest implementation does not allow sampling beyond the number of observations, even when sampling with replacement. Why is this? Works fine: ...
11
votes
1answer
12k views

Oversampling with categorical variables

I would like to perform a combination of oversampling and undersampling in order to balance my dataset with roughly 4000 customers divided into two groups, where one of the groups have a proportion of ...
10
votes
1answer
8k views

SMOTE throws error for multi class imbalance problem

I am trying to use SMOTE to correct imbalance in my multi-class classification problem. Although SMOTE works perfectly on the iris dataset as per the SMOTE help document, it does not work on a similar ...
9
votes
1answer
2k views

Oversampling: whole set or training set

I have a rather small dataset of 4 000 points (140 features) to feed to a NN binary classifier. The problem is only ~700 of them represent the second class. Is it more common to resample the whole ...
7
votes
1answer
11k views

Oversampling in logistic regression

I was trying to find out whether an oversampling can really make a model better. On this blog page, it says that it can improve a decision tree, but it shouldn't improve a logistic regression. ...
7
votes
1answer
4k views

Normalization/standardization: Should one do this before oversampling/undersampling the data or after?

When working with imbalanced datasets, should one do one-hot encoding and data standardization before or after sampling techniques (such as oversampling or undersampling)?
7
votes
0answers
2k views

After oversampling/undersampling is it always appropriate to adjust probabilities using the odds ratio regardless of the sampling method used?

I have an imbalanced dataset where the target class is <1% of sample. I apply oversampling or undersampling e.g. https://github.com/scikit-learn-contrib/imbalanced-learn. I run random forest on ...
6
votes
1answer
1k views

Oversampling correction for multinomial logistic regression

When modeling rare events with logistic regression, oversampling is a common method to reduce computation complexity (i.e., keep all the rare positive cases but just a subsample of negative cases). ...
5
votes
1answer
5k views

Problem with classifier after using SMOTE to balance the data

We've ran into a problem while training a classifier on an unbalanced data set. The response is binary with 0 indicating 'non defaulter' and 1 indicating 'defaulter' (it's a credit scoring task). ...
5
votes
1answer
463 views

During oversampling of rare events, why are the beta coefficients of the independent variables not affected, but only the intercept?

I have followed the King and Zeng paper and understand the consistency of the prior correction after oversampling in logistic regression. But I am trying to understand why the beta coefficients of the ...
4
votes
1answer
427 views

Classification of Huge number of classes

I have a dataset of samples belonging to >100 classes. I want to classify and/or cluster these classes. I have the following questions: 1) Is one classifier efficient for such problem? or one ...
4
votes
1answer
59 views

Does it makes sense to perturb images on-the-run while training CNNs?

I'm training a convolutional neural network by adding small perturbations (like rotation and shifting) to the images each time it gets a batch data. I think a better way of doing this could be to ...
3
votes
2answers
225 views

What if I factor the training set?

In pratice, it is usual that we don't have enough observations to build our desired models. An idea come to my mind is that the population can be factored: in other words, we can simply duplicate ...
3
votes
1answer
927 views

Is oversampling done for Cox regression data?

I have a dataset consisting of about 48000 people, about 40,000 of which die before the end of the study and get a failed = 1 and the remaining 8000 have a failed = 0 because they're either lost to ...
3
votes
2answers
86 views

Using patient characteristics to predict disease and confront with per hospital real cases

We have a large, nationwide prevalence study about the prevalence of healthcare-associated infections (HAI). We need to see if hospitals have more or less HAI than expected by their patients' ...
3
votes
1answer
1k views

General question on oversampling

I once read (unfortunately I cannot find the source anymore) that oversampling is only "allowed" when the class distribution is very, very skewed, i.e. 1:99 or worse. So in my case I have a class ...
3
votes
0answers
237 views

Run time of SMOTE function in package DMwR

I have a dataframe with 930 000 rows and 220 variables. The objective is a binary classification but my response classes are imbalanced. (88% - 12%) I want to use SMOTE to artificially create ...
3
votes
0answers
4k views

Should SMOTE oversampling be done before or after holdout validation's training/testing split? [duplicate]

Originally, without SMOTE, my ML learning steps go like this: Feature vectorization split data into X_train, X_test, y_train, and y_test use X_train and y_train for machine learning predict/test ...
3
votes
1answer
89 views

Proper Sampling - can I collect a two-group sample this way without issues? [duplicate]

I need to collect a two group sample for a comparison analysis (perhaps using logistic regression). The population that I need to extract a sample from is all firms from country A with activities in ...
2
votes
2answers
233 views

(When) Does simulating a larger sample from a small sample yield better results?

I am trying to classify a set of $p$ predictors into 5 classes. But my sample size $n$ is rather small, so I fear I won't get a very robust estimate. Now an idea would be to subset my data for each ...
2
votes
2answers
760 views

Balance classes when sampling

I have a large dataset describing numerous customers' behaviour and I am trying to solve a binary classification problem with a null accuracy on 90% (90/10 distribution amongst the two classes). ...
2
votes
1answer
630 views

Does oversampling affect sensitivity

I learned that sensitivity is not affected by oversampling since it identifies the proportions of correct classification for positive events. However, I also learned that by doing oversampling of ...
2
votes
1answer
51 views

Running SMOTE on unbalanced data

I've read a few answers to similar questions that advise the completion of SMOTE after splitting the data set into Train and Test sets however, the documentation and other examples I've seen run SMOTE ...
2
votes
1answer
1k views

How to report the results of a cross-validation to a paper: Can I manually select the best results?

Good morning. With regard to the interpretation of the results of the N-fold cross-validation with over-sampling (SMOTE), I raise the question to write up the paper. Currently, I have performed 5-...
2
votes
1answer
276 views

Over sampling for minority classes

I'm a bit confused with the idea of over/under sampling. People have mentioned that rotations are a good technique. By rotation, do they literally mean rotate the image by a few degrees and then add ...
2
votes
1answer
87 views

Visualising individual observations in weighted data

Context: I have a survey where I have deliberately oversampled particular groups of respondents. I am aggregating these groups and therefore I am using measures such as the weighted mean to ensure my ...
2
votes
1answer
647 views

ROC and false positive rate with over sampling

I'm modelling a rare event (say 1 in 10000) and I'm using an over sampled train set to cross validate and train my model. I'm using ROC as a global performance metric but there are business reasons ...
2
votes
1answer
510 views

Oversampling methods for numerical data (regression)

There are many oversampling methods for categorical labels (for example SMOTE and Rose, etc.). But, are there oversampling method for numerical labels (the thing that I want to predict with my ...
2
votes
0answers
560 views

Logistic Regression Class Imbalance and the use of weighting and undersampling

I have been working on a machine learning model using Spark (binomial) LogisticRegression. The dataset has what I think is a high degree of imbalance - roughly 1% of rows are labelled as events. The ...
2
votes
0answers
83 views

What are the recommended over-sampling or under-sampling techniques to handle multi-class imbalanced datasets for machine learning? [duplicate]

Many techniques are for two-class classification. What is the recommended techniques to over- or under-sample multi-class datasets?
1
vote
3answers
8k views

Running XGBoost with *highly* imbalanced data returns near 0% true positive rate. Tried SMOTE and it did not improve much. What else can I do?

I'm using XGBoost on a dataset of ~2.8M records of hard drive failures, where less than 200 are tagged as failures. After cleaning, there are 11 features in this dataset. Below is my ...
1
vote
1answer
506 views

Effects of Oversampling on Linear Regression

We often see that oversampling is a useful technique to combat skewed classes. But there is not much discussion about linear regression. If we just duplicate the original data, and train the linear ...
1
vote
3answers
2k views

Does oversampling/undersampling change the distribution of the data?

I have an imbalanced dataset (10000 positives and 300 negatives) and have divided this into train and test sets. I perform oversampling/undersampling only on the train set since doing this on the test ...
1
vote
1answer
319 views

Random Over Sampling to handle Data Imbalance

Was reading an article on imbalanced datasets where the event occurs and look at balancing the dataset. In that article, the event records were 2% of the total records. The author of the blog ...
1
vote
1answer
435 views

Assessing classification error with synthetic oversampling (SMOTE, ADASYN, et)

Consider a situation, where there are two unbalanced classes (n1 < n2). Some standard statistical methods advise to use SMOTE (or similar) oversampling methods to balance classes and train a ...
1
vote
1answer
516 views

is negative log loss affected by oversampling?

I'm working on a multiclass classification problem where negative log loss is the evaluation metric. My initial train set and my static test set have similar class distribution and my validation (20% ...
1
vote
0answers
48 views

Will oversampling help with generalization (small imbalanced dataset)?

I have an imbalanced dataset (2:1 ratio) with about 60 patients and 80 features. I performed RFE + stratified cross validation to reduce the features to 15 and I get an AUC of 0.9 with Logistic ...
1
vote
0answers
38 views

When is oversampling preferable to undersampling and vice versa?

When data is unbalanced, that is, when the distribution of classes being predicted is very uneven (e.g. 90%/10% for two classes or 10%/15%/75% for three classes), many machine learning models have ...
1
vote
1answer
42 views

How do i re-train a final model after using oversampling?

I am a bit puzzled about the process of experimenting with a model and oversampling and then translating it to the final version of the model that will be used: I oversample the data (only the ...
1
vote
0answers
72 views

SMOTE and Lagged Observations

I'm doing a project about the effect of synthetic oversampling in a machine learning context (more precise SMOTE for the oversampling of the minority class of a highly imbalanced target variable). The ...
1
vote
1answer
723 views

Oversampling a multi-labeled data set

Given a data set where each individual data point can be assigned to more than 1 class (a multi-class, multi-label data set), are there any guidelines for calculating oversampling weights, i.e., the ...
1
vote
3answers
870 views

Binary Classification in Imbalanced Data; Oversampling and Imputation

Together with two friends I participate in a university course about data mining in R and we chose the topic of bankruptcy prediction. We started with some "clean" data found on an "In class" kaggle ...
1
vote
0answers
118 views

influence of oversampling on Semi-supervised multi-label learning

I have suggested a semi-supervised approach for the hierarchical multi-label classification task. I have included the MLSMOTE oversampling technique as a pre-processing step, and then evaluate the ...
1
vote
0answers
10 views

Oversampling to Create Correlated Feature from Existing Feature

For demo purposes in Python, I would like to create an array or pandas df column that shares a high degree of correlation with an existing array. This is not as easy as it sounds and I don't think it ...
1
vote
0answers
121 views

While dealing with imbalanced classes, to what extent can we upsample a minority class? [duplicate]

I have my training data with the following approximate distribution: Negative events : 90,000 positive events : 5,000 Training a model would require to oversample the minority class (and might also ...
1
vote
0answers
380 views

Oversampling using SMOTE leading to bad predictions on test set

I have a dataset with an imbalanced binary target. One class accounts for about 94 % of the target variable. I used SMOTE to oversample the minority class but after the oversampling step when I train ...